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ABSTRACT 

A developmental project created prototype 
individualized instructional materials for the Inventory Management 
(IM) Course which could be incorporated into the Air Force Advanced 
Instructional System (AIS) . Typical course segments and a final block 
of instruction were selected, and instructional aaterials were 
developed for these units. The materials were field-tested with 
individual students and revised; they were then used with a group of 
students, revised a second time, and used by several Air Forcetf^ 
training classes. Results showed that all students attained all the 
learning objectives, and that a moan student time savings of SS% was 
achieved. Student attitudes were favorable and no problems were 
encountered. It was therefore concluded that the individualized 
materials should be used in the AIS since they resulted in increased 
efficiency with no loss of achievement. (Author/PB) 
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SUMMARY 



Problem . 

The purposes of this study were to: (1) select 30 representative 
instructional hours from Blocks I, II, and III of the IM Course that 
typify those blocks; (2) develop Individualized instructional materials 
for those 30 hours as well ai.. the entire Block IV (60 hours) of the 
course; ,{3) try out and evaluate the materials; and (4) document the 
results, problems involved, and processes used to develop the materials. 
This study Is part of a larger project, the Advanced Instructional 
System (AIS). The problem identification and prototype materials 
development were directed toward the AIS. 

Approach . 

The IM Course was analyzed and typical segments were selected from 
Blocks I, II, and III. As a result of the analysis. Instructional media 
and strategies were selected, and materials were developed for these 
segments as well as for Block IV. The materials were tried out Initially 
on one or two individuals and revisions made as required. The materials 
were then tried out on a classroom of students. Normal AF routines were 
followed as much as possible. All students were given criterion 
referenced tests during the block and an end of block multiple choice 
test. Final revisions based on this validation group were then made. 

Results . 

The 20 students In the tryout passed all criterion referenced tests 
and achieved a mean block time savings of 47%. Their mean block grade 
was 82.8%. An additional group of 57 students used the revised materials. 
They also passed all criterion reference tests with a mean time savings 
of 55%, and a mean end of block test score of 88.6%. Student attitudes 
toward the learning materials and methods were highly favorable. No 
major problems were encountered. 

Conclusions . 

The Instructional materials resulted In improved student efficiency 
with no loss of achievement. Student attitudes toward individualized 
instruction were favorable. No significant problems were uncovered to 
pose any threat to the AIS. 

This summary was prepared by Arnold L. Hanson, Technical Training 
Division, Air Force Human Resources Laboratory 
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INTRODUCTION 



PROJECT BACKGROUND 

The IM Project Is part of a larger project, the Advanced Instructional System 
(AIS). The AIS will be a large scale» computer-based instructional system used 
for Air Force training and education. The AIS will be an Integrated training 
system encompassing course material, Instructional media, and a training man- 
agement system. Initial ei'fort Is currently underway to develop prototype 
Instructional materials for three targeted Air Training Command (ATC) courses 
which can be converted to run under the AIS. The IM Specialist course Is one 
of these courses. 

The purpose of the IM Project Is to: (a) select 30 Instructional hours of 
course material from Block I, II and III of the IM course which are represen- 
tative of the segments In these blocks, (b) develop individualized instructional 
materials in support of the 30 instructional hours and also for the last 60 
hours of the course, and (3) try out and evaluate these materials. The project 
consists of the following four phases: (1) analysis of the IM course and 
selection of course segments, (2) analysis of course segments and selection of 
media and instructional strategy, (3) preparation of instructional materials, 
(4) try out and evaluation of instructional materials. 

Of partlcualr importance for the IM Project 1q the development of prototype 
course materials which can be converted into a full-scale, computer-managed 
system. Therefore, the analysis of the IM course and the resulting selection 
of course segments considered this factor as well as the usability of course 
materials prior to operating under the AIS. 
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OVERVIEW OF THE INSTRUCTIONAL MATERIALS DEVELOPMENT PROCESS 

The IM Project required System Development Corporation (SDC) to select segments 
of the existing IM course for which individualized (self-paced) instruction 
would be appropriate and to develop individualized instructional materials for 
thetie segments. In addition, SDC was to develop a strategy whereby the modified 
course could be managed, and to provide cost information and cost estimates for 
a course materials production configuration. Since the new course materials 
and management system are to be converted to use in a computerized instructional 
system at a later date, SDC was required to provide material and management plans 
that are computer-compatible, SDC has developed a step-by-step process for accom- 
plishing these kind of tasks. That process, as it was applied to the IM Pro- 
ject, is described in this part. 

A graphic overview of the development process is provided in Figure 1. Each 
box in this figure is explained in detail in the text which follows. 

TRAINING CONCEPT 

SDC began the IM Project with a generalized training concept applicable to all 

types of training courses which have the basic objective of teaching performance 

of a job. The basic principles of this concept are as follows: 

Trainees should be required to perform tasks in accordance 
with the on-the-job procedure. 

Theory and terminology (as opposed to hands-on-task performance) 
will be taught along with related practical tasks. 

Tests should cause students to demonstrate job-related skills 
rather than, or in addition to, ^'knowledge about the job". 

Instructional material should be sequenced in the order in 
which tasks and sub tasks are performed on-the-job whenever 
possible. 
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Figure 1. Overview of Instructional Materials Development 
Process 
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STEPS IN THE DEVELOPMENT PROCESS 



This report describes each step in the Instructional materials development 
process foK the IM course. For each step In the process, there Is a section 
of textt 

At the beginning of each section of text, there is a flow chart showing the 
substeps of that step. Each second-level flow chart begins with the box for 
that step, as shown In Figure 1, and ends with the box for the next step. 
These two boxes, beginning and end, are drawn with heavy lines, as shown 
In Figure 2« 

STEP 1: ACQUIRE A L IBRARY OF JOB AND COURSE MATERIALS 

The first step in the development of Instructional materials is to acquire 
Job and course documentation and develop a filing and referencing system 
for it (see Figure 2), 



■M STEP 1 1 

Acquire A. 
Library 6t 
Job and Course 
Materials 



Substep Irl 
Acquire library 



I 



Substep l-i 
Establish Cetitral 
Pile 



I 



STEP 2: 
Esubllsh Course 
Segments to 
Be Trained 



- . ^ / ' _ - ■ , . 

Figure 2. Substeps in Acquiring a Library of all 
Job and Course Materials 
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SUBSTEP 1-1. ACQUIRE LIBRARY 

For the IM Project library, SDC acquired the following types of materials re- 
lated to the existing IM course and to the IM Specialist's Jobi 



• 


Speciality training standard 


• 


Course charts 


• 


Plan of Instruction 


• 


Student study guides 


• 


Student workbooks 




IM occupational survey report 


• 


Lesson plans 


• 


Programmed texts 


• 


Student handouts 


e 


Materials and documents used on-the-job 


• 


Performance tests and within tests 


• 


Applicable Air Force Regulations 


• 


Course audiovisual materials 



The specialty training standardi lesson plans » student study gu:^.des and workbook 
proved to be most useful in developing instructional materials. The Occupational 
Survey Report is normally an excellent guide to the tasks that are performed 
on*'the«job; however the Report for Inventory Management was outdated. 

SUBSTRP 1-2* ESTABLISH CENTRAL FILE 

All materials were filed using a simple numerical filing scheme* and a central 
file index list was prepared* The central file Index list is located in 
Appendix A. 



ERIC 



12 



STEP 2 1 ESTABLISH COURSE SEGMENTS TO BE TRAlKED 



SDC V4a rdquested to orelect 30 houta o£ Inatructlon from the exlatlng IH courae 
of 240 houra and to provide IndlVldualixad Inatructlonal matortala for thoae 30 ' 
houra and alao for the laat 60 houta of the cduraei (Figure 3 praaanta Step 
2 and ita Subatepa;) 

The 30 houra aalected vara to meet the following requirements i 

• Converalon to Indlvldualiied inatruc^ton muat not Increaae 

the nuuiber of Inattuctora or Inetructor houra raqulredi ; 

e Sagmenta aalected muat be repreaentatlve of the courae aa 
a whole with respect to akillsi knowledgea and media 
requlrementa* 

• Segmenta selected muat have content such that indlvldt^alized ' 
Inatructlonal materials developed for them could be readily 
adapted to computer aaaiatad Inatructlon ahd/ot a computeif- 
manage^ couraei 

When good courae documentation la not available aa a base froin which to delect 
tegmenta for individualized instruction! developers uuat work directly from, a >: 
Job/taak data base. The job itaelfi and all the taaks performed in the courae 
of the job» muat be analyzed in order to aalect tasks requirlti^ training and 
to establish the suitability of individualized Inatructlonal techniquea for 
training each task* In addition i required characterlstlce of tralHeeai 
both for the Job and for the course^ must be established from a thovough Job/ 
task analyala. For the IM Project, the lengthy ptoceaa of establishing such 
a data base and performing the analyaia waa not necessary* ..Exiating coutae 
fflaterlals arid interviews with Inattuctora and subject matter experta provided 
a sufficient basis for the selection of course segments for individualized 
instruction* 
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STEP 2x 
Establish Course * 
Segments To Be 
Converted 

Substep 2-'l 
Establish Basic 
Selection 
Rationale 

Substep 2*^2 
Review Materials In 
Central File and make 
Preliminary Selections 
Using Suitability 
Rationale 

Substep 2-3 
Select Final 
Course Segments 

STEP 3: 
Investigate Student 

Population and 
Training Environment 

Characteristics 



Figure 3, Substeps In Establishing Job Segments to be 
Converted to Individualized Instruction 
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SU^STBP 2-l< ESTABLISH BASIC SELECTION RATIONALE 

SOC developed a three^-atage aeleetlon rationale for the tM courde< f^or Stage 
One» three var.iablee were considered In developing a list of lessons potentially 
suitable for conversion to Individualized Instiuctloni namely i 

• Proficiency levels required to be attained In the course as 
specified In $TS645XO» 

t Average percentage of time spent doing a given task on the Job 
as derived from the IM otcupatlonal survey report*. 

e Lesson effectlvlty> estimated for the effect that conversion to 
Indlvldualleed instruction would have on student petformance# 
(Estimations were based on such factors as lack of need for V 
student interaction to perform a taski estimated cost of ptc^ 
ductlon of learning aldsi and enhancement of student performancei 
particularly for task-otlented subject matter.) 

A scoring system was devised in which each existing lesson was assigned a score 
ct 2eto» onoi or two on each of the three variables* lesson segments with a 
score of two foir each variable were selected for the Initial list of potential 
individual lied course segments. 

Vor Stage Two» SDC evaluated the potential candidate? selected In Stage One <>ti 
the basis of total skill and knowledge requirements as It had been Requested that 
the segments selected be representative of the course as a whole in terms of 
skills and knowledges* Segments with similar skills and knowledge requirements 
were identifledi overlapping areas of these requirements eliminated, and an 
abbreviated list of segments totaling 30 to 40 course hours and meeting these 
requirements was developed. 

During Stage Threei a final determination was made on the following crlterlar 

e Lack of dependence of selected segments on unselected segt^ents 

# Logical Interrelationship of selected Segments 

# Kepresentatlve requirements for media 
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Other rationales can be used for the seleotlon process > depending on the 
quantity and quality of existing course materials and upon course objectlvesi 
One useful cue for considering course segments for Individualized Instruction 
Is that of low teat scores which may Indicate trainee difficulty in learning the 
course material. When students » over a period of tlmd| have difficulty with a 
particular course unit or segment > individualized instructional techniques may 
Improve student performance if materials arft well designed. Instructional 
materials developers can use student test scores to determine which course 
units are most difficult) particularly for the best students. 

SUBSTEP 2-2. REVIEW MATERIALS IN CENTRAL PILE AND MAKE PRELIMINARY SELECTIONS' 

^ch lesson plan in the central file was analyzed and scored using the Stage 
One selection rationale. Figure 4 shows the score sheet; explanations 
of the scoring appear at the bottom; The score s^heet itself shows the source 
of the Information used to assign a score. All lesson plans with average 
scores of two were selected as potential lessons for conversion. There were 
44 course hours in this list. The Stage Two flAlectlon rationale was then applied 
to the 44-hour list, resulting in the omission of 12 course hours (3 lessons) 
due to their similarity in skill and knowledge requirements to other selected 
segments. Preliminary selections are shown in Tables 1 and 2. 
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I 


2 ^ 


3 


4 


, .5 


6 


BLOCK III - STOCK CONTROL 


HOURS 


STS 645X0 


OCCUPTATIONAL 
SURVEY REPORT 


CONVERTABILITY TO 
SELF INSTRUCTION 


; AVERAGE 
1 SCORE VAL'JE 


TASK PERFORMANCE 
LEVEL 


TASK KNOWLEDGE 
LEVEL 


no 

k| 

§ 




StocW Conti^nl LpvpIa 




2 


2 


0 






** • 


2. 


Initial Spates Support Lists 


2 


2 


2 


0 


1 






3. 


MILSTRIP Requisitions 




2 


2 


0 


0 , 






4. 


MtLSTRtP Related Products 


3 


0 


2 


0 


0 






5. 


Base Procurement 


6 


2 


2 


. 0 


1 






6. 


Due-Out Revalidation and Due- In 


3 


0 


2 


0 


0. 








Reconciliation 
















7. 


Materiel Control 


3 


2 


2 


I 


0 






8. 


Monitoring Priority Requirements^ 


5 


I 


2 


0 


2 






9. 


Receipt and Due-*Out Releases 


4 


2 


2 


0 


2 




2 


10. 


Stock Balance and Consumption Reports 


3 


0 


2 


0 


0 






U. 


Base Excesses 


3 


2 


2 


0 


0 






12. 


Shipments and Transfers 


A 


2 


2 


0 


0 






n. 


Document Control 


2 


0 


2 


0 


0 








Stock Control Functions 


7 


2 


2 


0 


0 






15. 


Manual Accounting 


I 


1 ' 


1 


0 


0 


2 





0 « The individual Is not required to have a task 
knowledge » performance or subject knowledge proflciency| 

1 The individual can: (a) do simple parts of a task; 
(b) name part8» toolsi and simple facts about the 
task; and (c) Identify basic factf; and terms about 
the subject* * / 

2 « The individual oan: (a) do most parts of the task; 
(b) name the steps in doing the task and tell how 
each is done; and (c) explain relationship of basic 
parts and state general principles about the subject* 

0 ■ Percent of time performing ta^k is less than 5% 

1 « Percent of time performing task is between 5 and 8% 

2 « Percent of time performing task is greater than B% 

0 » Individualized instruction is not an effective mode 
of Instruction. 

1 ■ Individualized instruction is as acceptable an current 
mode of instructlont 

2 « Individualized instruction is the most effective mode 
of instruction* 

Figure 4, Part of the Score Sheet for Preliminary Selection 
of IM Course Segments 
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Col« 3. Score Values: 

Col, 4* Score Values: 
Col. 5i Score Values: 



Ta1>le 1. Candidates for Course Segment Selection <44 Hours) 



BLOCK I ^ CEMERAL ANi RESEARCH 

Lesson 6» USAP Supply Manual (5 hours) 

Lesson 7« Mastor Cross Reference List (3 hours) 

Lesson 9* Cataloging Handbooks (3 hours) 

Lesson id* USAF Federal Supply Catalogs (12 hours) 

BLOCK II - ITEM ACCOUNTING AND MAINTENANCE iSlIPFORT 

\ 

Lessoi\ 4i Remote Operations (3 hours) 
Lesson 5< Supply Issue Procedures (6 hours) 

Lesson, 6* Repair Cycle and DIFM Control (8 hours) 

BLOCK XII - STOCK CONTROL 

Leason 9i Receipts and Due*-Out Release (4 hours) 



Tabid 2. Preliminary Selected IM Course Segments (32 Hours) 



BLOCK I - GENERAL AND RESEARCH 

Lesson 6« USAF Supply Manual. (5 hours) 

Lesson 10« USAF Federal Supply Catalogs (12 hours) 

BLOCK II - ITm ACCOUNTING AND MAINTENANCE SUPPORT 

Lesson h« Remote Operations (3 hours) 

Lesson 6» Repair Cycle and DIFM Cpntrol (8 hours) 

BLOCK III - STOCK CONTROL 

Lesson 9» Receipts and Due-^Out Releases (4 hours) 
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SELECT FINAL COURSE SEGMENTS 

SDC Instruct ional development personnel visited Lowry AFB, where the IM course 
Is given, to discuss final stlection of segments with course instructors and 
with subject matter experts. Selected course segments were audited. 

Discussions with instructors and subject matter experts and observations resulted 
in. the substitution of a 3-hour lesson for a 5-hour lesson. The 3-hour lesson 
had been on the Stage One potential list, but was not selected in §tage Two. 
Discussions with subject matter experts made it apparent that there was very, 
little difference in skill and knowledge requirements between the twp lessons. 
The substitution of the 'S-hour lesson for the 5-hour lesson allowed SDC to 
sslect exactly 30 hours of course segments. 

Based on the results of the preliminary course selection rationale and on the 
data collection and analysis effort conducted at Lowry AFB, a final selection of 
course segments was made. Table 3 lists the selected course segments and 
the associated Instructional hours. All course segments selected are logically 
tied togelher, are representative of skills/knowledges and media requirements 
of the IM course, are suitable for Individualized instruction, and can be ' 
implemented without dependency on nonselected course segments, 

Table 3. Selected IM Course ^;egments (30 Hours) 



BLOCK I - GENERAL AND RESEARCH 

Lesson 7. Master Cr'-ss Reference Liat (3 hours) 
Lesson 10. USAF Federal Supply Catalogs (12 hours) 



BLOCK II - ITEM ACCOUNTING AND MAINTENANCE SUPPORT 

Lesson 4. Remote Operations (3 hours). 

Lesson 6. Repair Cycle and DIFM Control (8 hours) 



BLOCK III - STOCK CONTROL 

Lesson 9. Receipts and Due-Out Release (4 hours) 
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STEP 3i INVESTIGATE STUDENT POPULATION AND TRAINING ENVIRONMENT CHARACTERISTICS 



Inetructlonal materials should be prepared at a level of difficulty appropriate 
to the reading leveli aptltude> and mental ability of the student population* 
The number of students per class » the physical environment In the classroom) 
and the adaptability of the classroom to audiovisual equipment to be used on an 
individualized basis mus^". also be liivestigated and findings Incorporated into 
Instructional materials design. Figure S presents Step 3 and its Substeps« 



The student population for the Inventory ^{anagement Specialist course ranges 
from "Project 100»000" personnel to college^level personnel. Most students are 
Just out of basic training and have no^Alr Force Job experience* The majority 
of students attain or exceed the minimum aptitude level score of Administrative 
60 or General 60. However » there are a number of students below this minimum. 
Further consideration is being given to lowering this minimum aptitude level| 
perhaps to 45. "Project 100>000" personnel are included in the course but are 
not identified as such. The "Project 100,000" designation only become apparent 
when failures are being considered for elimination or wash back to later classes. 

SUBSTEP 3-1. INVESTIGATE STUDENT APTITUDE AND EDUCATIONAL LEVELS 

For the IM Project » records of former students in the existing IM course were 
examined and course instructors were interviewed to assess the aptitude require- 
ments for the course and the aptitude range and average aptitude level of former 
students. Reading level and mental ability were assessed through instructor 
interview. IM course students are somewhat above the Air Force average in 
administrative aptitude and intelligence. Reading levels are Judged to be about 
those for high school seniors. However i because materials may be used in the 
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STEP 3! 
Investigate Student 

Population and 
Training Environment 

Characteristics 



Substep 3-1 
Investigate Student 
Aptitude & Educational 
Levels 



I 



Substep 3-2 
I>otennlne Student 
Load 



I 



Substep 3-3 
Oeterulne Classroom 
Characteristics 



\ 



I 



STEP 4 1 
Analyze Job 
and Lesson 
Contents 



Figure 5. Substeps in Investigating Student Population 
and Training Environment Characteristics 
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Z6ro**draft environment, In which there way be a shortage of high aptitude men, 
the decisions was made to write materials at a basic English level (8th grade) 
to accommodate all students. 

The Air Force administers certain standard aptitude testa to all alrnen. These 
provide some indication of the airman's learning aptitude and reading ability* 
The AFQT and Gen AQE are measures of general learning aptitude. AFM 35-8 pro- 
vides an explanation of these tests and their meanings. An analysis of ^hese 
scores for potential students in a course provides an estimate of the range of 
abilities of the students and some idea of the relative number of "good" or 
"less apt" students. It is Important to look at this kind of information In 
planning the instructional materials. If the required Information about the 
students is not available, it may be necessary to provide some testing program 
or other data gathering means to obtain tha information. 

PRL-TR-66-1, Estimating Reading Ability Level from the AQE General Aptitude 
Index, gives a method for estimating the reading level required for a training 
course If aptitude score levels of current and former trainees can be obtained. 

For certain kinds of courses, it may be i very important to determine student age 
range» socioeconomic class, or sex» since for some kinds of material to be 
learned^ student learning styles and past experiences are quite varied. Within 
the Air Forcei age range is small and many classes are all-male. More research 
will be needed before materials can be developed that effectively account for 
socioeconomic class differences. 

SUBST^ 3-2. DETERMINE STUDENT LOAD 

Instructors in the existing course were interviewed to determine current student 
load for the IM course. It is Important to determine the number of students 
expected to enter the course and the percentage of those expected to complete 
it in the normal length of time or within an extended length of time. Instruc- 
tional strategies, methods, and media are selected with reference to this 
information* 



During the fiscal 1972 period, a new class of 100 students is scheduled to enter 
the course each week. The aptitude range of students during this period is 
not expected to ch nge, but the aean aptitude level can be expected to decrease 
vlth more students In the lower range of scores. 

For these reasons, it is desirable to write the Inventory Management lndlviduall«ed 
Instruction at a basic English level. However, the necessary technical language 
; required on the job will be included. 



jWith a. new class each week there will be a class in each of the eight weeks of 
/the present course at any given time. Therefore, tryout of Individualized 
instruction will be possible at any time, 

SU8STEP 3-3. DETERMINE CLASSROOM CHARACTERISTICS 

Instructional materials requiring audiovisual equipment must be designed to be 
compatible with the physical layout and equipment potential of the classroom. 
Therefore, classroom size, layout, electrical connections, equipment, and equip- 
ment budget must be investigated before materials design can begin. SDC found 
for the IM course that classrooms could accommodate audio tape players and slide 
projectors to be used on an Individualized basis, as long as student load Is 
optimally controlled and scheduled. 

The Inventory Management Specialist course is taught in a single air-conditioned 
Structure and utilizes 16 classrooms (eight remedial training classrooms, how- 
ever, are available in another building). Each claasroom is equipped with a 
felt board, black board, screen, and overhead projector. One of the 16 class- 
rooms is equipped with six operational remote keyboard printers. The maximum 
student capacity for each classroom is 25; however, the desired optimum size 
for traditional 'instruction would be 10-15 students, depending upon the subject 
matter. Two sets of tape recorders (Sound-o-matic I), slide projectors (Kodak 
AV900), and 16am projectors (Graf lex) are available on a scheduled basis. 
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STEP A» AHAl^YZB JOB AND LESSON CONTENTS 
Thei purposes of this step are to: 

# Select tasks to be trained • 

e Identify skills, knowledges, attitudes, and tools/equipment/ 
inaterlals associated with each task to be trained* 

♦ ■ Establish the sequence of behaviors by which tasks are performed. 

• Identify standards and conditions for successful task performance, 
(see Figure 6 for presentation of Step 4 and its Substeps.) 

SUBSTEP 4-1. RESEARCH JOB AND EXISTING COURSE MATERIALS IN DEPTH 

Training is most efficient when it is directed toward preparing students 
to perform the tasks required of them on the job. For this reason, developers 
of instructional materials should plan on spending at least ten percent of 
their time and resources on analysis of the Job and of existing lesson 
materials. 

When a course already exists, as it did for IM, and the developer^s task 
consists primarily of individudlization of existing material, task analysis 
can be focused on a precise analysis of existing lessons. New course materials 
can be derived from the existing course materials, provided that the existing 
materials are up-»to-date and cover job requirements adequately. 
The first activity performed in a task analysis is to inventory all tasks 
performed on the job. A task Inventory lists each task in the same format: 
task/verb first, noun and descriptors second. An example is: Turn vernier 
knob counterclockwise. The completed task inventory is then used to guide 
preparation of task analysis information forms and to select tasks for 
training. - 



STEP 4: 
Analyze Job 
and Lesson 
Contents 



Substep 4-1 
Research Job 
and Existing 
Course Materials 
In Depth 



I 



Substep 4-2 
Select Tasks 
for Training 



Substep 4-3 ' 
Prepare Task 
Flow Charts 



Substep 4-4 
Prepare TAIS 



I 



STEP 58 
Develop Learning 
Objectives 



Figure 6, Substeps In Analyzing Job and Lesson Contents 
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For the IM coursei SDC used two paper forms to record task analysis Information! 
These two forms — a Task Plow Chart and a Training Analysis Information Sheet 
(TAIS) — vjere selected for use because SDC found that the current coarse 
materials did cover performance of all the tasks required on the Job. When 
course materials of this kind or detailed task analysis information are not 
available^ an additional form titled? Task and Skill Analysis Sheet (Piguto 7 ^ 
can be used to record Job-related information), 

SDC examined existing course materials » Job descriptions , specialty standards, 
and occupational surveys to gather all of the information required to prepare 
an Inventory of tasks > Task Flow Charts and TAISs, In addition, SDC observed 
the current IM course and interviewed instructors. 

SUBSTEP 4-2. SELECT TASKS FOR TRAIN 

To select tasks for training, SDC prepared an inventory of IM tasks, within 
selected lesson segments, using the Occupational Survey Report and Specialty 
JTra inking- Standard. ^JbJi ta^ks wuM thiSn^amlnei fftiuslmlljirljty oljUTOceduxPi « 
skills I or knowledges, and for any group of tasks similar In these respects, 
one was chosen as representative of the group and was therefore selected for 
training. However, when instructional personnel for the current IM course 
reviewed these selections, they informed us that the Occupational Survey was 
outdated and did not Include all tasks upon which students would later be 
tested for promotion* In addition, the amounts of time spent on various tasks, 
as listed in the Survey were no longer accurate. Therefore the personnel 
assisted SDC In making a new selection, which is reflected in the Task. Flows, 
discussed in Substep 4-*3i 
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1 TASK 


AND SKILL ANALYSIS SHEET 


1 TASK NAMES 


fverb) • (object) 


i TASi^ rn Nrt. 


1 TASK TYPE: 




1 TASK LOCATION: 




1 PnilTPVRMT/PORMS TDl . . . 


1 PERFORMANCE TIME: 


FREQUENCY: 


1 NO. PERSONS: 


COORDINATION: 


1 TASK CRTTICALITY: 




1 TASK LOCATION: 


/ 


1 RTAi^TTNfi CXlRi 


1 ENDING CUE: , ^ 


1 Trtni<!. _ ': 


1 TEST EQUIPMENT: 




1 MiTPBTAI.S! . ... 


1 ENViaONMEiilJ 




1 BEHAVIORS : 




1 HAZARDS: 


1 ClIPPDUT«TnM. 


1 PROCEDURES DOCUMENTS: 




1 OTHER 1 




1 BPOPnuMANr.P. RTANnARD! 


1 TASK STEPS: 




1 1. 




1 ^' 




1 3. 




1 ^' 




1 ^' 
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Figure 7. Task and Skill Analysis Form, used to Record Job-Related 
Information 



SUBSTEP 4-3. PREPARE TASK FLOW CHARTS 

Task Flow Charts represent the tasks to be iralncdi their task elements i uad the 
relationship among them* A Task Plow Chart was prepared for each lesson for whlcli 
Individualized materials were to be prepared* Decisions were made during this 
Substep as to the sequencing of tasks within a lesion and any changes to be made 
to organization and sequencing of lessons in the existing course* 

Fifjure 8 shows a Task Plow Chart for the IM course* The numbered items In 
this chart correspond to the explanatory paragraphs below* 

TASK FLOW EXPLANATION 

Lesson Topic Name^* SDC tn on earlier step determined this 
topic was stjltable for. conversion to Individualized Instruction. 

Block and ,Lej}json ,f Dj , These Identification numbers refer to 
the planned sequence of Instructional materials. 

Task Sta teme nts * Each of the numbered statements refers to 
a task performed on the job that is covered wltliln this 
lesson topic* * ^ ^ 

M aj or Ta sk Flow * Each of these boxes corresponds to one of the 
numbered task statements* Their sequence indicates the order In 
which the tasks are performed and are to be taught* 

^A^.k Element s_ /.^y * ^ach of these boxes represents a sub task 
of a major task* Their sequence Indicates the order In which 
the task elements are performed and are to be taught* 
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If there is no existing course for which new instructional materials are being 
preparedi SOC recommends that instructional developers prepare Task Plow Charts 
for the Job and also for* the course material. The first set should relate 
directly to the tasks performed on the job. After Task and Skill Analysis sheets 
have t>een prepared, tasks required to be trained have been determined t and 
teaching sequence of major blocks has been determined » a second set of Task 
Flow Charts oriented toward course presentation sequencing can be preparedj 
using the first as a reference for sequencing task performance • 

SUBSTEP 4-4 • PREPARE TAXS 

A Training Analysis Information Sheet (TAIS) is shown In Figure 9, The 
circled numbers correspond to the explanatory paragraphs belov. 

TRAINING ANALYSIS INFORMATION SHEET (TAIS) EXPLANATION 

TAIS IDs « Lesson module and block number are the same as 
for the associated Task Flow. A TAIS Is prepared for 
each task shown on a Task Flowt therefore » Task Code number 
on the TAIS refers to the associated task number on the " " 

Task Flow. 

Task* The task statement from the Tat^k Flow is shown here. 

Conditions . Conditions are, in the case of the IM course, derived 
from existing course and IM Specialty standards. They state what 
must be learAed and in what context performance must be demonstrated. 

Standard . This is that performance standard that is considered 
adequate to ensure that the task is learned under the stated 
conditions. 

Task Elements . Each of these statements corresponds to one of the 
task elements shown on the Task Flow Chart. Each is a subtask to 
the task for which the TAIS is written. 
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BLOCK NUMBER l_ 



LESSON ^fOOULE B 



0 



TASK CODE 04 
(^^_J^ESSON NUMBER 10 



TRAINING ANALYSIS INFORMATION SHKET 
TASK: Obtain Index Numbers from YSHs 



'2. CONOITIONSt Given four (4) Federal Stock Numbers, obtain the assigned 

index numbers using the Federal Supply Catalog Identification 
List (C-1610-IL-AF) 



^ 73. STANDARD: No errors 
U, TASK ANALYSIS: 



TASK ELBWENTS 



0 



1. Locate FSN. 



PREREQUISITE KKOVLEDOE 
OR SKILL REQUIREMENTS 



© 



la. Know numerical 
sequence. 

lb* Reads off assigned 
number. 



TRAINING 
MATERIALS 



. Q. 



C-1610-IL-AF 



REFERENCES 



C-l-AF, C-1610- 
IL-AF, 

C-1610-HL-AF, 

SG/WB-I-10 

HOI-IO 

HOI-10-1, 

HOI-10-2, 

C-RL-l-AF, 

Transparencies 

ill20, 121, 122 

123 



Figure 9. A TAIS for An IM Task 
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Prerequisite Knowledge or Skill RequlrementQ i These requirements 
are listed for each task element # Each roust be taught or known 
before training on the actual task begins* 

Training Materials * These are materials (other than SDC-produced 
materials) that are required to perform the task. 

References * These are references to materials prepared for the 
existing training course and for the job* They were used by 
SDC in developing Hae individualized materials* 

Training Analysis Information Sheets provide basic Information for developing 
Learning Objectives and for developing the instructional materials themselves. 



STEP S : DEVELOP LEARN I NG OBJECTIVES 

Learning objectives are the base from which course materials are producedi 
Le a rning obJective&. specif y ti\e kind ot performance to be teated, also. Two 
kinds of learning objectives are developed: (1) criterion objectives, and 
(2) enabling objectives. Criterion objectives are end objectives associated 
with a specific task. Each specifies student behavior required on the Job* 
Enabling objectives are subobjectlves to criterion ebjectivesi each enabling 
objective represents a skill or knowledge necessary to successful performance 
of a 'given task* (Figure 10 presents Step 5 and its Substeps*) 



0 

0 
0 
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STEP 5t 
Develop Learning 
ObJectlvoB 






SubstGp 5-1 
Develop Criterion 
Objectives for 
Each Task 







Substcp 5-2 
Develop Enabling 
Objectives for Eech 
Criterion Objective 



i 



STEP 6: 
Develop Test 
Items 



Figure 10. Subsceps In Developing Learning Objectives 
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Figure 11 ahowf a sample Learning Objectives Worksheet for a specific IM Task. 
The circled numbers match those In the explanation below. 

LEARNING OBJECTIVES WORKSHEET EXPLANATION 

Itgarning Objective IDs . These Identity numbers are carried over 
from the IDs for the task. Each TAIS has a matching Learning 
Objectives Worksheet. 

Crite rion Ob.lectivee . Oncj or more criterion objectives for the 
task are listed here. 

^) Enabling Oblecttvee. Enabling objectives for the task are listed 
hare. 
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BLOCK NUMBER 



LESSON 1«>DULE B 



.0 



0TASK CODE 
LESSON NUMBER 



04 



10 



LEARNING OWECTIVBS 



CRIlRfllOK OBJECTIVES 



©1. Given various FSNs, look 
up the index number In the 
FSN to Index Cross Reference portion 
using C-1610-IL-AF. 



ENABLING OBJECTIVES 



©1. State FEDERAL STOCK 
NUMBER when questioned con- 
cerning what information must be 
known for direct use of a USAF 
Federal Supply Catalog. 

©2* State INDEX NUMBERS 
as being what the FSN is 
cross referenced to. 

0 3. When asked how the two 
types of index numbers are 
identified within the Item Identlfi^ 
cation List, fill in NUMBER and 
PART 1 for complete item ^ID and fill 
in LETTER and PART 2 for minimum 
description. 



Figure 11. A Learning Objectives Worksheet for the IM Course 
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SUBSTEP 5-1. DEVELOP CRITERION OBJECTIVES FOR EACH TASK 

What qualities do good criterion objectives have? The following qualities were 
considered essential to development of the IM instructional materials i 

• Each objective requires the student to demonstrate a skill or 
knowledge that is directly related to task performance, Such 
•skills and knowledges can be derived from the TAlSs previously 
prepared. The conditions section of the TAIS provides the 
Information required. 

• Since test Items will be derived for each learning objective, 
observable behaviors that can be tested must be specified as 
learning objectives. 

Criterion objectives, if there is more than one for a task, should be listed 
in the order in which skills and knowledges are required in task performance. 

SUBSTEP 5-2. DEVELOP ENABLING OBJECTIVES FOR EACH CRITERION OBJECTIVE 
Good anabling objectives specify in a step-by-step fashion the elements of 
bshavior- involved In reaching criterion objectlvaa. Enabling objactlvsa should 
possess the same qualities as do criterion objectives, but on « »or«*deiralled 
level. 

STEP 6: DEVELOP TEST ITEMS 

Test Items are keyed directly to criterion learning objectives. A test item 
for an objective is phrased to show how it can be known whether a Student has 
or has not met the learning objective. Criterion test items are written foe 
each criterion objective. Enabling teat items (subtest items to a criterion 
teat item) are developed to teat for mastery of each element of a given task. 
(Figure 12 presents Step 6 and its Substeps.) ^ 
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STEP 6^ 
Develop Test 
Items 







Sybstep 6-1 
Develop Criterion 
Test Items for B&ch 
Criterion Objective 



Substep 6-2 
Develop Enabling 
Test Itens for Each 
CrTter Ion Test Item 



f 

STEP 7 X 
Select Instructional 
Methods and Media 



Figure 12; Substeps in Developing Test Items 
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Figure 13 shows an example of a Test Item Worksheet « The numbers In circles 
match the numbers In the diecussion below* 



TEST irms EXPLANATION 

Teat Item Worksheet IDs ♦ These IDs match those on the TAIS and on 
the Learning Ob^lective Worksheet » * 

Cri terton Tests » At least one item is written for each learning 
objective. The statement labeled CONDITIONS on the TAIS is used 
to derive the content and context of the test Item. 

Enabling Tests . These items test the individual elements of skill 
or knowledge that are required for successful performance on each 
criterion objective. 



SUBSTEP 6-1. DEVELOP CRITERION TEST ITEMS FOR EACH CRITERION OBJECTIVE 
The following qualities should be built into criterion test items s 

• Good criterion test items require students to demonstrate skills 
and knowledges directly related to the criterion objectives • 

• Good test items elicit observable and measurable behaviors. One r 
should keep scoring procedures in mind when developing test items* v 

For paper and pencil tests » questions with fill-in answers are 
often prefevable to multiple-choice questions because the 
behavior required to answer is usually more closely related to 
actual task performance on the Job. However, if multiple-choice 
questions are used, they should preferably have four or five 
alternative answers. 

Performance*oriented (as opposed to paper and pencil) items ' 
should include a statement of the context in which an observation 
is to bo made of student performance^ the exact behavior(s) to be 
observed, correct performance, and potential incorrect alternatives. 
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BLOCK NUNBES _ 
LESSON MODULE 



TASK CODE 
1 I LESSOR NUMBER 



0 



10 



TESTS 



CHITERION TESTS 



0 



FSNS? 
a. 
b. 
c. 



1. What are the Index 
Numbers for the following 



1610-546-3319 — (BOOl-0272) 
1610-081-7184 — (0096-8000) 
1610-301-1557 — (8000-9686) 
1610-996-0036-BD — (BOOO-2234) 



ERABLlNO TESTS 



km 



What Infomatlon tbud^t be 
knovm for the direct uai of 
a USAF Federal Supply Catalog? 
(Federal Stock Number) 



0 2. T] 
refett 



Numbers) . 



The FSN la cross 
ferenced toj (Index 



There are two kinds of the above 
numbers within the Item Identifica- 
tion List. The complete Item IP is 
one and the minimum descriptlbri Is 
the other. 



0 



The flrjst character df 
the minimum, description is a 
( letter ), lite minimum description 
is known as a Part (2) list. 



0 



4. The complete item ID is 
known as a Part (1.) list and 
Its first character is a ( number ) , 



Figure 13. A Test Item Worksheet for the IM Course 
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SUBpTBP 6-2^ DEVBtOP ENABLING TESt ITEMS FOR EACH CRITERION TfiST ITEM 

to write enabling test Items » developers must analyze the criterion item In 
depth to determine each separate skill and knowledge Involved In performance* y 
The TAIS and the list of enabling learning objectives provide the Information 
used to derive these elements. Enabling items muj^t be constructed Just as 
carefully as are criterion Items, to elicit observable, measurable behavior, 

STEP 7t SELECT INSTRUCTIONAL METHODS AND MEDIA 

Pour major factors determine Instructional methods and media to be used 
for individualized instruction: 1) characteristics of the tasks to be 
trained, 2) training concept and instructional strategy, 3) characteristics 
of the students, and 4) physical realities, such as classroom space, equipment 
budget, etc. In Step 7, a worksheet is made, showing each task's and each 
task element's major characteristics Insofar as methods and media select-tons 
are concerned. The worksheet is then used with types of learning list and 
a selection matrix to Identify the optimal method-media combination for each 
learning element listed on the TAIS discussed in Step 4. The workshc types 
of learning list, selection matrix, and the selection procedure, are based 
upon the work of personnel at the U.S» Army Signal School, Fort Gordon, Georgia. 
References to this work are provided in Appendix B to thlfl >locument. 

Signal school personnel feel, based on review of methods and media selection 
literature and upon their own practical experience In developing Instructional 
materials, that prime methods and media selection factors aret a) student 
characteristics, b) general instructional strategy characteristics, and c) type 
of learning involved for each task and task element to be learned • their selecti 
matrix encompasses all of these factors* 

(Figure 14 presents Step 7 -and its Substeps.) 



STEP 7: 
Select Instructional 
Methode and Media 



I 



Subatep 7-1 
Frepate Methods and 
Media Worksheet 



I 



Substep 7-2 
Utilize Selection Matrix 
to Select Optimal and 
Method Medium for each 
Task Element and Lesson 



I 



Substep 7-3 
Determine Media Devices 



I 



STEP 8! 
Specify Instructional 
Strategies 



Figure 14. Substeps In Selecting Instruction^ Media 



SUBSTEP 7-1. PREPARE METHODS AND MEDIA WORKSHEET 

This substep has several parts. First, developers must decide upon the nature of 
the tasks elementSi the nature of the student population, and the instructiortal 
strategy based upon the general training concept • These decisions are entered on 
the Methods and Media Worksheet. CFigurc 15) 
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The numbers in circles in Figure 15 match the numbers in the discussion below. 

METHODS-MEDIA WORKSHEET EXPLANATION 

0 Ta8k Codes ♦ The task code number and the task element number 
are listed as the identification code, one to a line. 



© 
© 



0 
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© 
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Task and Task Elements » Statements are listed as they appear 
on the TAIS. ^ 

types of Learning . The Types of Learning List, Figure 16, 
is used to select a one-^word description o^ learning type for 
each task as a whole and for each task element* The explanations 
on this list are used to make the determinations « Examination of 
the learning objectives and test items for a task aids in making 
the determinations. VThen type of learning has been selected, it 
is entered on the Methods and Media Worksheet, Column 3« 

Methods Class Code, On the right of the Types of Learning List, Figure 
16, are two columns, one for Methods, and one* for Media. When Learning 
Type for a task or task element has been determined, the Methods Class Code 
shown is transcribed onto the Methods and Media Worksheet, ^Column 4. 

Media Class Code. The Media Class Code shown on hhe Types of Learning List 
is also tr;in.9crlbed onto the Methods and Media Worksheet, Column 5. 

Self-Paced. This is the first of the instructional strategy 
decisions to be made to aid in selecting methods and media. To 
fill in this column for a particular task or task element, the 
following list pf questions is answered} 

• Does the learning include value Judgements? 

• Does the subject material or equipment change frequently? 

• Must the student compile the learning within a ^^^ctured 
time frama? 

• Will a self-p Aced program be more costly than a group-paced 
program? 

• Does the task element require a team effort? 
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TYPES OF LEARNING LIST 



LEARNING TYPES. Select the most appropriate 
learning type for each task element first* 
Then select the type that best characterizes 
the entire task. 



!• Factual Infonnation « Learning names, events, 
termsi definitions. 

P^ciston ^ Selecting a course of action 
from several choices. 

3» Diagnos is > Using cues to identify solutions 
to problems or to define a problem* 

4. Procedure > Carrying out a sequence of acts 
or operations in the correct order* 

5. Principle . Knowing theory, concepts, rules} 
understanding why, how, or x^hen events occor. 

6. Organization . Knowing how parts are orga- 
nized functionally. 



7. 

8. 

9. 
10. 
11. 
12. 



Construction . Knowing how objects are 
stvuctur,ed. 

Identification . yKnowlng names and locations. 
Operation . Knowing how someth^.ng functions. 
Inspection . Checking against standards. 
Manipulation . Manually working with objects. 
Ev aluation . Making value Judgments. 



METHODS 
CLASS 
CODE 



IV 



V 
VI 



MEDIA 
OlASS 
CODE 



A • 



B 



C 
D 
E 
F 
F 



Figure 16. Types of Learning List 
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If the answer to any of these questions Is Yes, group Instruction 
Is more appropriate and an "0" is entered in the Self-Paced columns. 
If the answer to all the questions is Noi then a ^^1^* is entered in 
the Self-Paced column for the task or task eletienti 

Experience . Does the trainee have previous training or prior 
experience which Is applicable to a given task or learning element? 
If 80, a "1" is entered in the Experience column; if not, an "0'* is 
entered > This determination is made by first considering whether 
the students have had training or experience before entering the 
class. Then for a given task or task element /learning type 
combination, consideration Is given to whether there has been a 
previous related task or task element, possibly of a different 
learning type. 



© 



0 



Reeponse-'orlentad . If the trainee nuet respond to the ta.k or 
learning element by doing sonethlng, the answer 1« yet, and a. "1" 
la entered for thla column. If the trainee nu.et only read, aee, 
or hear the content, then the anawer la No and a "0" Is entered. 

Learning Element Method . To fill In thla column, the Methoda and 
Media Selection Matrix la used. Figure 17 ahowa this matrix. 
Thgmatrix hfts its own nianbered aet of explanations. Read number 
(ij anrt \ 2j for a deecriptlon of how Learning Element Method 
entry is obtained. 

Q-^ Learning E lement Media . See Figure 17 and read its explanation 
~ number (3J for a description of how the Learning Element Media 
entry is obtained. 
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lesson Method * Once all at the task ana learning element metnoutf 
are selected using the selection matriXi a general or combination 
method for the entire lesson can be entered by Inspecting the 
method entries for all tasks and elements of the lesson and 
determining all o£ the methods required for the lessoni 

Lesson Media e Once all of the task and learning element media 
are selected using the selection matrixi all the media required 
for the lesson can be listed in the lesson media column* If 
more than three media . are required to be used vithin a 1-hour 
lesson time blockp the developer should consider reorganizing 
the task structure to avoid thlSi or should consider making all 
learning elements vithin a lesson more compatible with respect 
to the Response-Oriented entry. Generally » many changes of media 
within a short time period disrupt learning and increase training 
t ime « 
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SUBSTEP 7-2 » UTILIZE SBLBCTIOK MATRIX TO SELECT OPTIMAL METHOD AND MEDIUM 
FOR EACH TASK ELEMENT AND LESSON 

The Selection Matrix is ehown lu Figure 17 • It ie ueed to make methode and 
media selections to enter on the Methods and Media Worksheet. The numbers 
in circles on the matrix match the circled numbers in the explanation belov» 



1^ Instructional Strategy . For a given task or learning element /the 
pattern o£ aseros and ones on the Methods and Media Worksheet match 
as a pattern on the Selection Matrix » The method and media 
appropriate for the task or task element are selected from the 
matrix row with the matching pattern* 

MethodClass Code * For the same task or task element considered 
in Tl j above/ the mskthod class code on the Methods and Media 
WorksKeet matches a method class code on the Selection Matrix* 
the matrix box that is the intersection of the instructional 
strategy pattern (row) and Methods Class (colximn) contains an 
abbreviation that stands for the selected method. Explanations 
of the abbreviations are at the top of the matrix. The method 
is entered on the Methods and Media Worksheet. 



3^ Media Class Code . Media selection is performed in the saM way 
06 methods selection. The selected media' are enetered on the 
Methods and Media Worksheet. The media listed in the matrix 
were derived through rational analysis and through analysis of 
research findings relating media use to learning efi^ectiveness. 
Hew research may require changea in the matrix at a later date, 
and perhaps other instructional strategy factors or types of 
learning will be added for consideration in the selection process. 



4^ Abbreviation Explanations. These phrases explain the abbrevia* 
tlons contained In the body of the matrix* 



SELECTION MATRIX EXPLANATION 
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Figure 18 dhows a portion of the Methods and Media Worksheet that has baen' 
prepared for the IM course* 

{ilotlce that the I^esson Media column Includes the use of a tape recorder In 
one Instancei although the Selection Hatrlx does not recoi^mend the audio 
medium for students who can read and comprehend faster than they can listen 
and comprehend* SDC added the use of the audio medium to lessons in which 
one of the learnlni^ types was Manipulation so that students could perform 
the manipulation without having to look away to print i For manipulation 
tasksi students probably cannot read faster than they can listen* Another 
portion of the Selection MatrlXi not Included in this report « contains methods 
and media selections for this clrcumstance» and an audio medium is specified* 



t 
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After filling out the selection natrlxi some consideration of practical con<- 
atralnts is Important! These constraints may include limitations In facilities i 
equipment or personnels This can be either In the actual classroom or In the 
support areas* Such considerations may limit the course developer*8 options but 
they also enable an early avoidance of Impractical decisions* If a constraint 
Ifiv^oses minor limitations which do not degrade Instruction there are no real 
problems. If a constraint threatens to degrade the Instructlonar systeroi then 
a problem has been identified which should be posed to those who have the 
authority to effect the solution* 

The selection of media devices Involves many factors beyond the scope of this 
report and although use of a particular device seldom Imposes real limitations 
on the ability of learning materials to teach» its selection does require that 
the materials are physically compatible with it. This requires careful attention 
to the many details that affect the match of film, tapA> etc* to the device and to 
classroom or maintenance environments. This coordination should be started even 
before the final selection of device or media to preclude subsequent problems that 
interfere with student learning and create real time delays or expensive correc- 
tional procedures. 

SUBSTEP 7-3. DETERMINE MEDIA DEVICES v \^ 

A specific medium, as defined on the Selection Matrix^ does not require a 
specific device for display of training material. For ex-^mple, print ,cbuld 
be in a booki on loose sheets of paper, or on microfiche. ^>'* 

For the IM course, practical realities narrow down the f?osslble choices of 
devices for presentation of materials, but the bael^ media are available 
written word, etlll pictures, instructor lecture, audio, or actual objects. 
Motion visuals present a more difficult problem, as would commercial or 
specially-designed teaching machines If they were required. At a later time, 
the IM course will very likely be computer-assisted. Materials produced now 
on paper, or possibly even on slides, may then be displayed at a computer 
terminal. But at this time, that mode of presentation Is not available. 
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rot Indiyldualized instruction, film and instructor lectures «ra not as aatls- 
(ACtoty AS slide sets and audio tapo, because it is expensive to individually 
pace filne and lectures. They are nuch more cost-effective when viewed and 
listened to by a group** Individual film devices are quite expensive. For the 
IM course* for any motion visuals required, the use of audio-pulsed slides 
will be explored. The tasks for which motion visuals were specified are quite 
simple and therefore the elides can replace motion visuals without a loss in 
learning efficiency. 

For some applications, the number of audiovisual devices required may make the 
coat of audloviaual devices prohibitive. The number needed depends on the 
proportion of the courae that la to be taught using the devices. If an Indi- 
vidualized course' utilizes only audloviaual oethoda, then of course one device 
will be required for each atudent. For the eight-week IM courae, which 20 
atudenta enter each week, It was determined that 25 devices should be aufflelent. 
However, the proportion of time that students spend with the devices Is small 
relative to total eatimated individualised course time. A student in the IM 
course will spent only about 5-lOX of the time ueing an audiovisual device, 
but at the beginning of the l«st 60 hours, which are eelf-pacad, about 20 
atudenta will be wanting these devicea at the sane time. It was decided that 
acquisition of 25 devicea would allow the placement of at leaat one device 
for each two student? in the room in which the firat two modules of the self- 
paced portion of the courae would be taught, ensuring that delay time would 
be relatively abort. A emaller number of devicea could then be placed in 
e«ch of the rooma used for later modulee utilising audiovisual devices, aince 
eeif-pacing will apread atudent progreas. It is hoped that research studies 
In this area will aoon provide courae develop^^ra with a decision-making tool 
to estimate the number of devices required for given numbers of studente and 
given proportions of courae time in which the devicea will be utilised. 
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, To determine practical media devices for a couraei developers must Investigate 
money avallablsi cost of devlcesi and compatibility with the physical chArac* 
terlsclcs of the claasroom» and student load* These Investigations ehould take^ 
place Very early In the materials development process In order to ensure com* 
patlblllty of materials and devices* 

STEP Bi SPECIFY INSTRUCTIONAL STRATEGIES 

To determine Instructional strategies to be used In developing the IM courasi 
SDC designed a worksheet to be tllled«-ln as development work progressed. Theni 
during Step 8| this worksheet was completed* ThH completed worksheet specifies 
the instructional strategy bas^s with which all materials must be compatible* 
A program of instruction reference sheet for the course was also prepared for 
Step 8i (Figure 19 presents Step 8 and its Substeps.) 

SUflSTEP 8-1. PREPARE STRATEGY WORKSHEET 

Figure 20 shows the Instructional Strategy Worksheet, fllled-ln for the IM 
course. The top item on the sheet specifies the given Instructional strategy 
conditions under which the IM course Is to be developed. The Air Force expects 
SDC to fulfill these conditions. The next group of items specifies SDC's 
general training concept under these conditions. The final item specifies 
an organizational strategy to optimize learning effectiveness. 
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STEP 8'j 
Specify Instructional 
^ Strategies 



Substep 8*^1 
Prepare Instructional 
Strategy Worksheet 



Substep 8-2 
Prepare Program 
of Instruction 
Reference Sheet 



I 



STEP 9: 
Develop and Product 
Instructional and 
Supporting Materials 



Figure 19. Substeps in Specifying Instructional 
Strategies 
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On the worksheet In Figure 20, each of the numbers in circles matches a 



requested to provide Individualized materials and to select course 
segments for which Individualization was possible. Computer adapta** 
blllty was also requested In the contract. This condition effects 
the form in which student responses can be required. Course materials 
requiring an "out-loud^' response^ a manipulation of an object^ or a 
long written passage should be curtailed as much as is feasible If a 
course is to be presented later by computer peripheral equipment, 
since computer peripheral equipment is not cost*-eff ectlve in evalu-- 
ating these kinds of responses. SDC is not producing the entire IH 
course. Therefore, the organlMtlon and content of SDC-produced 
materials must not disturb the unity of the course as a whole. ATC 
Reg 52-3 describee policies for development of learning objectives, 
progress checks, and measurement tests. 

Individualization for the IK cgurse Is defined as self-pacing. Other 
aspects of Indivldualltatlon that were considered and rejected are 
production of different materials for different student learning 
atyleBt such as all audio materials for those who read the poorest, 
and all print materials for high achievers In school, and the use of 
different language levels for identical content, with the student 
selecting the level he feels most comfortable with or with a pretest 
determining the most appropriate languoge level for a particular 



2^ Task/Knowl edRe Orientation . This item refers to the educational 
goals for the IM course. A Job training course, such as the IM 
course, should require the trainee to demonstrate his ability to 
perform tasks associated with the job, not merely his knowledge 
about the job. 




student. 
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INSTtlUCTlOMAL STRATEOV WORKSHEET . 



Ilndlvldualigfttlon of ttateri«>la> self-pacing. 



Contract Condltlon9 « List those affecting instrtictlonal strategy r 

0 

CotDputer Adapatftblllty ■ 

Ciouree eeymenta omet dovetail with aeaaents not betfftg produced 
by SDC/ < V . .' " ' " . ' . . ■ ■;"^V\'''' 

Meaaurement and progreee check atrategiaa must conform to 
ATC R eg i2^3i Maaeurement of stwllent Progreaa and AcniWemant# 

:.j ^ l^Z^ , ^ .v.. ■ ' ■ ' ■ . J."/. ^ - 

General Training Concept for Courae t 



0 



2 j Taek/Knowledge Orientation . Check concept appropriate for 
course. 

not task oriented. Knowledge only. 

some segments require task, orientation or 

problem solving. f 

J{ course trains for a Job| most segments task oriented. 

other i explain^ 



Figure 20. Instructional Strat^y Worksheet for the IK 
Course (Sheet 1 of 6> 
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Teetlnjfc * The IM courae trains students to perform taskSi but these 
tasks are primarily paper and pencil tasks i rather than equipment* 
or peoples-related tasks* Therefore, realistic testing requires paper 
and pencil tests oriented to task performance. 

Materials Sequencing * There are many vays course materials can be 
organized* When the goal of a course is to elicit successful task 
performance the best way to sequence maV'^rial is by the order in 
which work is performed by task flow« This can be done within 
each task for the IM course i and among tasks in some casesi But 
some IM tasks are not performed In a specific order* These tasks 
are to be sequenced as they were In the original IM course, to 
enhance the compatibility of the new and old materials. 

Role of Instructor * Since the IM course is to be self-spaced, the 
Instructor will not be the prime means of presenting material* He 
will^ therefore I have more time for performing the other instructor 
roles listed* 

A dvance and Review OrRani^^grs * The use of organizers is recommended 
for the IM course* An 4(iv\ance organizer is presented prior to 
lesson content and provides education materials relevant to the 
learning objectives of the lesson laodulei but not essential to 
their achievement* Each advance organizer Introducea the student 
to the subject matter tb^bi^^resented and Identifies the skills 
and knowledges to be learned* For material with more than a 
uinimal degree of cognitive content as (opposed to psychomotor 
content) studies show that advance organizers enhance learning by 
aiding cognitive organization. A review organizer is a restatement 
of the advance organizer and Is presented after the lesson content 
and just before the criterion test. The student re-reads the 
organizer to assist in deteriiiining whether he can meet the learning 
objectives of the lesson* 
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^^ Teatini^ i Check concepts mo3t appropriate for courses 

JC •'hands-on** testing on objects* 

all paper and pencil testlng> oriented to knowledge* 

X paper and pencil testingi oriented to task 

' performance* * 

f , ■ " 

^ other> explain. 

0 Material Sequencing s Check concepts appropriate for most 
bnlts. 

s 

simplest to most difficulty no other organizer* 

X by units of content) topics* 
X by task flow. ; 
otheri explain* 

Role of Instructor . Check usual roles. 

prime means of presenting material. 
X diagnose student skills and knowledges* 
X monitor student progress * 
X assist students upon request. 
^ ^ other I explain. 

Advance and Review Organizers , Check appropriate statement* 

X recommended. 

not recommended. ^ 




Figure 20. Instructional Strategy Worksheet for the IM 
Course (Sheet 2 of 6) 

er|c 



•i 

Presentation of StlittuluS i This entry specifies the cue and prompt 
strategy to be employed with i&oet material « Descreaslng cues and 
prompts are very effective In producing learning. At the end of 
a cue and prompt sequence, the learner needs no prompts In the 
stimulus presentation in order to make a correct response* When 
material Is not too dlffi^cult to learn and remember* limited 
prompts will be satisfactory. 

Student Response . This entry sp'gclfles the forms In which responses 
will be elicited from trainees. Selective responses ate true-f else i 
ttultlple^-choicsi matching » etc. Constructed response Items more 
closely approximate IM tasks than do selective response Items. 
Multlple^cholce answers do enable material developers to branch 
students more easily and will be used for remedial branching as 
well as for knowledge-'orlented items. The methods used to elicit 
etudent response In the Dt course will be varied also to provide 
student Interest. Many responses of the same kind can become 
boring to the student, free responses are not computer adaptable 
in that the correctness must be evaluated by an instructor ^ who 
must also provide feedback. 

Feedback . Research has shown that Immediate feedback to the student 
about thk correctness or incorrectness of his response is most 
effective In producing learning. There are clrcuostancea and course 
c<$))tent where i.hls is not possible but Imnedlate feedback is nost 
appropriate for the IM course. 
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PrM6tttatlop St It^uluB < Check api^roprlate method* 



straiRht texV> no prpmpca. 
X decifeaeitia cuea and prompts to responee* 
X . : limited promptdi 
■^Y-;.^.. Other I explain • • ; : 
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8^ Reai>onse » Check appropriate, typea » . 

X aelective reapQnae; 
X conatructed reaponae» 

free reeponeei ^ 

otheri explain* 

Feedback . Check appropriate concepta, 

X Imedlate feedback f or each^ reaponaei 
. feedback at enda of tpnlta} unite are 



otheri explain. 



Plgure 20. Inatructlonal Strategy Itorkaheet fojr th« IM 
Course (Sheet 3 of 6 ) 
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Student Progresglon t A cost-effective method of guiding a student 
through self -paced material Is usually linear. That Is, a student 
progresses from one lesson element to the next In the sequence • 
Bach student responds to each item. SDC plans a modified main-line 
track in which the majority of students will achieve a corract 
response for each main-line track stimulus presentatloni Howeveri 
jlf correct response is not obtalnedi then branching to supplementary 
material will be performed. The combination of these two methods 
is often called linear eclectic* Some of the supplementary materlai 
will not be prepared until after the validation phase of the IM 
Project. At that time, developers of the materials will have more 
information about the learning problems a student is likely to have 
as he progresses through the material. Supplementary materials can 
then be constructed to solve these problems. 

Rev low . Use of a review element at the end of each lesson is a cost- 
effective method for the student to enter himself in a loop to pre- 
vlously studied material to ensure that he achieves lesson learning 
objict Ives* 

Choice of the '^no review** category would cause a great deal of 
supplementary material to be developed) based on diagnosis and 
evaluation, for each leseon. To avoid this, th^ materials will 
aid a student to self --prescribe review based on his responses to the 
criterion Items of a leseon. A review orge&.tzer and student review 
matrix will be Incorporated in the materials for each lesson for 
this purpose* A review matrix provides the student with references 
to materials to be reviewed for each learning objective. 
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10} Student Pto&t»a^ion > Check usual means. 

X linear I ^ 
X branching i sttall step » 
. branching > scrambled book* 
other t explain* 7 

Reviev * Check appropriate concepts* 

no review • ' 
X review element at ends of lessons* 
^ other, explain* 




figure 20. Instructional Sitdtegy Worksheet for thi IM 
Course (Sheet 4 of 6) 



Evaluatlon « Evaluation Is here defined as referring to any measure- 
ment and diagnosis ocher than feedback to responses on Individual 
items* SDC decided not to use any pretest prior to the course or 
to any given lesson. Students generally have no background exper-- 
lence with the subject natter that could be tapped with a pretest* 
Measurement of student performance will occur at the end of the 
course^ for a course grade and also at the end of each leaaoni in 
the form of a criterion test« The diagnostic type of evaluation 
will be used within a lesson to assess student performance on enabling 
objectives and also after each criterion test to aid the student in 
deciding whether he should progress to the next lesson or wash-back 
to the prior lesson* * 

Language Level i Appropriate language level for the IM course is 
Adult Basic because of the varied student abilities in the course* 
Adult Basic in about 8th grade reading levels with the addition of 
required terminology for the tasks to be trained. 

Cognitive Level . (Adapted from Taxonomy of Educational Objectives i 
Benjamin S« Bloom» Ed* See references in Appendix of this document,) 
The IM course does not require students to demonstrate the higher 
Intellectual abilities (analysis » synthesis and evaluation) because 
the Job l9 highly proceduralized and forms are used to organize 
work. Therefor.e» the IM course need only require students to apply 
what they have learned in the course to Job performance* 
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BvAluatloi!^ . Check appropt late types and circle locatlonst 
X cliaghcetlQ type» iocatt6n A B 

other type • explain and ll8t loc$tlon 

Lpc A * prior tb beginning courae 
^c B • prior to lessor elftwenta 
Loc C * withiii lesson 
Loc D ■ subsequent to lesson 
Log E » subsequent to ell lessons 





13^ Language Level . Check appropriate ieyel* 

X basic for adults* 

college level. 

. other > explain. 

Cognitive Level . Check highest level of most material* 

. knowledge 

^ ^comprehension 

JC application 

analysis 
• . syntheses 
evaluation 



Figure 20. Instructional Strategy Wbrksheet fgr the IM 
Course (Sheet 5 of 6) 



Course Presentation Locatlon i Instructional methods and media to 
be usedi plus student load and classroom characteristics » determine 
where lesson materials can be used by students • For the IM course i 
no special facilities i such as movie theaters » on--the->Job locations i 
or simulation facilities » are required. Students can uae progtemmed 
instructional materials anywhere* Student rooms and lounges can ' 
be used for tape^-sllde presentations If students can be allowed to 
check out equipment « 

Course Organization * The chart depicts the way In which SDC Intends 
to organize the IM course* 
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is) C6ttr yif »ient ittioi^ Lt^c^t ioa i Chtck £6ailble locationsi 
X biMftt<K>ttyitbotitoieyt 

X at fame > - ' V ' ' " ''^ \ ^ \ " 

apadlil facility taqulradi Explain for which klnda 



of mauriali* 



16} Coursa Oraanltatlon » Show graphically and/or descrlbai 



Bloc|ca I'-llt 



5 Leaaon Modulea 



Block IV 



8 LeaaoKi Moctulaa 



Each laaaon nodule contalnai 



Advance 
Organlaat 



Les8on . 
Blamant 



Leaabn^ 

Reviaw 

Ble|aant 



^Crltatlon; 
Taat 
Biankant 



.Evaluation 
Element 



An End-o£-Block Teat will be pr^^pated for uae at the end of 
Block IV. ■ : ' ■, . ' ; - ^ . m; , 

Legendi 

Block • A niaj or block of instruction • • 

"*Le8aon Hodule A 8elf*-contalned unit and the amalleet atan4-* 
alone component of the cour0e* / 

Advance Organiser Introduction to atudenta ot organizing 
^concepta and of the pktUa and knowledge tp be ptesent^d 
in the ieasoni element » ' 

Laaapn Element Stimulua^ te8ponfle> faed^cik^equeh^cer^^^^^^^^ 
V eluded enabling progre^^ checka per ATC Reg j52'*3. 

teaaon Review Element Revle;^ orgMlter a^ at^^ 

Criterion teat Blement n teat for maiitaty of 90% learning 
\ oblectlveaieatiafiea raqulr progteaa 
checkai piftr ATC Reg 52-3 • 

Byaluatibn ElemWt « Dlagnb for remediate Inetructlontf fot, ^ 
atudents who have not maa tared a maduleVa leartilng bbjectlye* v 



Figure 20 r Ins true tional St rat egy^^^W^ for the IH 

Couvse (Sheet 6 of is) 
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SUBSTEP 8-2, PREPARE PROGRAM OF INSTRUCTION REFERENCE SHEET 

Using the Infonoatlon from the materials produced In Steps 1 through 7 and 
the Instructional Strategy Worksheet i a Program of Instruction Reference 
Sheet Is prepared* A portion of the IM course Program of Instruction 
Reference Sheet Is shown below in Figure 21« Instructional materials 
developers can then receive a work package consisting of the Program of 
Instruction Reference Sheet i Task Flowsi TAISs» Learning Objective WorksheetSi 
Test Item Worksheets » Methods and Media Worksheets) and the Instructional 



Strategy Worksheet to use in producing Instructional materials* The Program 
of Instruction Reference Sheet serves as an Index' to work package contents # 


PROGRAM OP INSTRUCTION 




REFERENCE SHEET 




PHASE I Selected Lessons from Block I# II, III, 


• 


Lesson Module At Master Cross Reference List 




Tasks t See Task Flows to 




See Training Analysis Information Sheets 


^to ♦ 


Objectives: See Training Objectives Worksheets 


to 


Test Items t See Test Item Worksheets to 




Method & Media: See Methods and Media Worksheet Task 




Codes to • ' 




Strategy t Strategy Worksheet i all items 




Lesson Module B: Federal SupdIv Cataloj^ 




Tasks: See Task Flows to . 




See Training Analysis Information Sheets 


to 



Figure 21. Portion of the IM Course Program of Instruction 
Reference Sheet 
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STEP 9> DEVELOP AND PRODUCE. INgtRUCTIQNAL AND SUPPORTING MATERIALS 

Instructional materials are prepared In accordance vlth the Instructional 
strategy worksheet /using the reference materials specified on the program 
of Instruction reference sheet* For tha IM course i each of the substeps 
described below was performed in sequence for each module of instruction* 
Several persons performed this step, each writing a group of modules, so 
all module drafts were sent to a technical editor who evaluated and revised 
the PI material as ne<;essary, and to a publications editor who revised where 
necessary for consistency of style and format in the total package. (Figure 
22 presents Step 9 and its Substeps.) 

SUBSTEP 9-1. PREPARE INSTRUC^ONAL MATERIALS FOR EACH LEARNING OBJECTIVE 
m> TEST ITEM 

The Methods and Media Worksheet specifies the basic instructional raethbd 
and media for a given module. For the IM course/ all modules utilize 
programmed Instruction as a basic method, with some modules also utilizing 
practical exercises. Media utilized are actual objects, tape recoilder, 
slide projector, and of course, print « 

^ ' - . ■ ■ . . ' ■ • " ' ■ • • f 

Materials developers began their production of the instructional materials 
by preparing lesson element materials first. To do this, the relevant 
task flow, TAIS, learning objectives worksheet and test items sheet for th^ 
module were reviewed. The developers then took the roU of the IM student 
and worked through all of the current instructional materials Just as the 
Student does. 

A series of PI frames were written for each learning objective of the lesson 
module. Drawings were rough-drafted, photos were copied, and these materials 
were inserted where appropriate. Humor and drawings were utilized where 
appropriate in the materials to enhance Interest. Material from early tasks • 
was utilized in subsequent tasks to reinforce responses learned earlier. 



STEP 9i 
Develop and Produce 
Xnttruotlonal «nd 



1 



Sub8t6p 9«*1 
Prepare Instructional 
Ihterials for Bach 
Learning Objective & 
Test Item 



Substep 9-*2 
Prepare Instructor 
Materials 



I 



Substep 9-3 
Prepare Student 
Instructions 



Substep ^4 
Prepare Measurement 
Test 



Step 10 
Prepare ;tnstructlonal 

MiMawfat f tin 



22. Substeps in Developing Instructlooiil 
and Supporting Materials 



Ady^nct OrMnlzer Materials 

Advance organizers were prepared after lesson element production* For some 
lessonsi advance organizers are entirely print « For others » tape and slide 
or tape presentations constitute the advance organizer element, depending 
on the media specified on the M^.thods and Media Worksheet « 

An advance organizer presents basic concepts and procedures required for 
task performance. The advance organizer ia designed to be viewed or read 
immediately before beginning the lesson element. It can be viewed again as 
part of the review element, prior to the review quiz. A portion of an 
advance organizer that utilizes the print medium Is shown in Figure 23* 

Tape and slide presentations were produced In-house by SDC. the simpler 
slides were prepared by the programmed Instruction frame writers utilizing 
ordinary photographic equipment! Some slides were prepared by SDC's 
Cartographic Department, primarily those requiring greater artistic talent 
than waa possessed by the instructional developer. Tape presentations were 
prepared utilizing a good quality cassette recorder. ''Draft** versions were 
prepared utilizing a variety of untrained voices. Once content is reviewed 
and edited to optimize learning, an SDC employee with professional experience 
in movies and commercials will do the final narration. 

It is beyond the scope of this report to discuss how programmed instructional 
materials should be written, or what constitutes a good frame. There are 
many good books available on the subject, however. A short list of books 
useful to beginners is contained in the References, 

Lesson Element Materials 

Lesson element materials were written first. 

The lessons are in PI format. A lesson covers tasks to be performed, functions 
equipment and materials involved, purposes of the task, and basic concepts and 
vocabulary. 

Figure 24 shows an example of the programmed instructed format. 
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li AF FORM 2005 



GO TO 9:2 



2. MANUALLY, BASE SUPPLY 

Sometimes, the head of a department will make a turn-^ln for the whole group. 
This person is called a Propert; !| ^ Custodian* The propetrty custodian completes 
another form before the AF form 20U!> Is filled out* This other form is 
fllle<J out by tho propoerty custoditin before he makes the turn-In at Base 
Supply, 

A ■ 

3, Look at Figure 5» What form will the property custodian fill out if the 

shovels being turned-ln come from him? AF form . ^ is filled out 

by the property custodian before the shovels are* turned-in to the Base 
Supply, 



Property Custodian 





Figure 5, 



4. AF FORM 60Ib i« correct. 

Even though the property cuetodien filled out en AF form 601b, you still 
nuet complete an AF form for the items he turned in. 



Figure 24. Portion of a Lesson Element 
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Review Eleroent Mateglals 

Review element materials were produced thirds h review element starts, with PI 
framed that direct the student to review the advance organizer element » and/or 
review lesson element subjects of the student's choice » and then take the 
review qul2i A subject review page lists the lesson's subjects and specifies 
their page numbers within the lesson element/ The review quiz consists of the 
enabling test items from the Test Item Sieet. Answers are given at the end 
of the quiz I along with page/step references for review, if answers are Incorrect • 
A portion of a review quiz is shown in Figure 25 « 



The questions that follow aummatize this lesson* Your task la to read each 
question and fill in any blanks with the ap ropriate word or words/ or make 
the correct response to a multiple-choice question* You may use, if necessary^ 
applicable publicatlon(s) to aid you in answering the questions* 

Complete the entries without referring to th^ lesson text* At the conclusion / 
of thene questions you will find a scoring key which you can use to check the 
correctness of your answers. 

1 * Identifying Procedures for Processing Receipts or Turn^Ins 

1*1 Requested items that have been shipped from either requisition 
source or another source ave called RECEIPTS ^ 

1*2 to receive unrequested items from a source of supply la termed an 
. shipment « 

1*3 If the equipment came from a property custodian » then he must 
have first filled out: 

a* DD Form 1348-1 

b. AF Form 601b 

c. AF Form 2005 

d. AFTO Form 350 

1«4 What paperwork bust acconipany a DIFM item when it is being turned 
in to supply? Pick the letterCs) of your choice* 

a* Part I AFTO Form 350 

b. Part 11 AFTO Form 350 

c. Copy 1 of DD Form 1348-1 



Figure 25* Portion of a Review Quiz 
73 



Ctlf rlon T»tt Element Material! 

Parfo^net ttats contain the cvlterlon teat Itema Crom the Teat Itan Sheet 
for the laaaon. Figure 26 ahowa a portion of a performance teati 





wnAC ar^ cna two geAarai tourcM for Itictiitlng bate iupply aatatt? 








a« Requisition 




be Transaction 




Ce Expandable Iten 




di Turn- In 




iUkii vn 




When property la returned to base supply* you nust fill outi 




aa DD Form 1348-1 




b. AF Form 601b 




Ce AF Form 2005 




de CC 67-80 




e* AFTO Form 350 » 


4. 


What Is done when a partial receipt is processed* 




Se Do nothing 




be Fill In the DD Fora 423 Interim sheet 




Ce Modify the exletlnt due^ln card 




de Output a new due-ln card 




e. Hone of the above 


5. 


For each requleltlon a receipt card should be preoareda 




Returnlna property to baae supply Is termed a 



^Figure 26 e Portion of a Performance Test 
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Evaluation Elemen^. Mi^teriaU 

Bacaude evaluation materials are used by the Instructor to aid the student ^ 
they were prepared as part of the Instructor Materials production process 
dlscuaaed In Substep 9*-2« instructional Materials for each Learning Objective 
for the IM Course are shown in Table 4» 

SUBSTEP 9-2* PREPAkE INSTRUCTOR MATERIALS 

An Instructor Guide was prepared for each lesson* This guide consists of 
three basic parts; a listing of required equipment and training mat erlald, 
a sequential list of lesson events and related Instructor actlvltlesi and 
th4 evaluation element for the ledson* The evaluation element cotislats of 
a scoring key for the criterion testi instructions on documenting teat 
results, and several possible prescriptions for the student to follow^ 
depending on criterion test score attained* . The prescriptions direct the 
student to begin the next lesson , repeat the current lesson, or review 
specific subjects in the current lesson with tho help of the instructor* 
A portion of the instructor Guide la shown in Figure 27* 

■ ■ ■ ' ■ ' ' \ . 
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AIS IHSTRUCTOR GUIDE 

COURSE TITLE J Inventory Management Specialist LESSON MODULE i C-'Remote Operations 

EQUIPMEHT TRAINING MATERIALS 

Slide Projector Slide/Tape Package 

Tape Recorder PI Package 

1066 Remote Keyboard Printer Remote Package 

Student Schedule 



Lesson Events 


instructor Activity 


1. Slide/Tape Presentation 

2, PI Text 

3* Controlled Practical Exercise 


Assist student 1 upon request* to Initiate 
slide/ tape presentation (presentation 
length! IS minutes) 

On Demand 

Instructor Demonstration: 
t 1 1 Form Loading 

111 Open upper cover 

1.2 Insert first sheet of DD Form 
1348-*! face dovn 

1.3 Lift out-'of •paper svitch 
lA Advance form under sprocket 
l.S Release paper balls 

1,«6 Turn manual feed-out knob to 

advance forms to first line of print 
lt7 Snap paper bails back into place 

rri Setting Vertical Tabulator 

2.1 Pull knob out as far as It will go 

2.2 Turn knob clockwise until one of 
the form out stops le directly 
under the detent 

2.3 Push knob back In place 



Figure 27. Portion of an Instructor Guide 
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SUBSTEP 9-3 • PREPARE STUDENT INSTRUCTIONS ' 

Student Instructlond were prepared to accompany eac u lesson* They are located 
at the beginning of the lesson booklet. Instructions Include those for using 
programmed Instruction materials, for operating tape and slide equipment , 
for starting and stopping the media presentation^ and for beginning the lesson 
element! The instructions are in PI format^ The student has the option to 
branch past the Instructions for using PI matarlals and for operating equipment 
If he has had previous experience. Part of the first page of the student 
Instructions for one module Is shown In Figure 28. 



3. Since you» the student, are usually called upon to make some type of 
reaponsei it would be advisable for the steps which fallow to be coveted 
until you have artived at a decision^ 



Ai Take e piece of paper ot cardboard sufficient in size to nask (or cover) 
these following steps. 

now move this paper down to 
the next step of lines and 
proceed to the next step*, .. ■ _ 

t ■ . 

5, The most descriptive tern for the use of this piece of paper would 
be e 

a« step 
b. line 
c* mask 
d. bill 

t 

STOPt 

00 ON TO THE NEXT PAGE 

: ' . ' P 



CCput the top edge of 
.the paper here) • i 




Figure 28. Portion of Student Instructions 



j|t)|dTBI^ M« . PREPARE MBASUREK^ tEST 
A ili^ •^^•u^«^At t««fe w«i pif«p«re4 £oF use at the end of Block IV, the 
ii^i eegwent Of the IM course. Biock iv contains eight leM^ 
ell' 0f > vhtch Were individualited. The tteasuremerit U«t , c^ an End-of- 
jlock test, covers these eight Xesson wduies. Sbdehtrinust sco^^ 
the test to graduate from the course. The EndTOf-BlofeJc test Was prepari 
to be co«vatible with the new lesson asterials, A portion of it 1« shown 
■in Figure 29, .,, . ' ■ ,/ '^.'v^ 



JABR64530-l^ , BLOCK IV ACADEMIC TEST 

1, Desks and typewriters are examples of itens with ERRC designators i 

a, XB2 and NU2 

b. XB3 and XD2 

c. ND2 and NP2 

d, XP2 and XP2 

2, Which organisation has primary responsibility for validating a request 
for equipment authorisation? 

a, i Storage and Issue Section 

b. Demand Processing Unit 
. c. Request Validation Unit 

d. Allowance/Authorisation Unit _ 
a. Any organisation that gets stuck with the Job 

3, To make sure an allowance document is c|frrent, you would look Irii 

a. APR 0-10 

b. MEMI 

c. ASC Listing 

d. Guide to Allowance Documents 

e. Modern Health 



Pigure 29. Block IV Academic Test 



STEP 10 1 PREPARE INSTRUCTIONAL MANAOKHENT PLAN 



Management of self-^paeed indlvlduaU^ed t^ralnlng courses la more complex than 
le management of a more traditionally structured course • When a course is 
self-paced I course and lesaon completion times vary with the individual • The 
time that a student will be ready for job assignment cannot be predicted with 
cettalnty» The role and schedule of the instructor Is affected also« An 
Instructional management plan is needed well before the new self*»paced 
materials are in official use In order to anticipate and resolve it^jnagement 
problems I to enable the training school staff to familiarize themselves with 
their new roles and duties, and to set out a general plan by which training 
will be accomplished* 

Instructional management planning is actually and iteractlve process during the 
Instructional materials development phase* Detailed plans for many factors 
that must be considered in the plan require coordination with the training school 
staff both administrative and instructional personnel and Joint agrees 
ments that are compatible with constraints on training scnool resources must 
be reached.' A considerable amount of informal research and analysis may be 
required in the investigation of possible schemes for scheduling students 
and allocating resources. For example, the question of where trainees will 
be physically located while using the instructional materials must be addressed 
In the plan* Planning personnel might initially propose that all lesson 
materials be physically located in one "check-out" room, only to find that 
the classrooms used for instruction are so dispersed physically that students 
would be spending a half hour a day checking materials in and out* 

Of course, when Instructional management planning personnel are physically 
located at the training school, are familiar with current management procedures, 
and have access to the adriinistrative and instructional personnel whose duties 
and roles will be affectedl by implementation of the plan,' problems like this 
diminish. If planning personnel are not located at the school, several 
visits are required to gatl^er management planning information and to identify 
constraints and llmitationEl tha^. will affect planning* 
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For th« rtviied IM couvta, Inttruc^lonal m«iuis«i«nt wlU at flrat ba manual, 

aa U tht aanatMiant of tha currant IK eouraai Latari a computav ayatam taay 

h% avtilabU vhleh can aid In uAny of tha nanai\«aant taakai Tha inatructlonal 

MatfMwnt plan mat ba aultabla for in«tructtoaal nanagmnt bafora tha . 

CMputar tyataai la oparatlonali 

Fltura 30 Qhowa tha substepa involvad In praparlni an inatructlonal ttanafwant 
plan* 

8UBSTBP 1(W. IBmCFT HANAOEKEHT FUNCTIOMS fOR THE IM COURSE 

Tha firat atap In davalopaant of an Inatructlonal wmagaaant plan la to Identify 
tha aaiasanant functlona that mat ba parforaed within an Inatructlonal ayataa that 
aaploya lodlvlduallt atlon of matarlala and aalf-padng of atudanta. For tha IM 
. couraa, SDC axaalnad docuaantatlon for aavaral auch ayataaa and conatruetad a llat 
of functlona that could ba conaldarad typical for thaaa •yataaw. Tha functlona 
can ba llatad In four aajor catagorlaai 1) functlona related to atudent place- 
■ent, echedullng, record keeping and report prepaiatlon} 2) functlona releted to 
laetnictlon, preaentatlon of Mterlala, Monitoring progriaa and dally counaellng 
of etudeatei 3) functlona related to tettlng, aaaetalng atudent progreaa, and 
planning of the etudent'a learning actlvltlea} and 4) functlona related to 
evaluating and revlalng the couraa and analysing atudent perforaance, 

The flret coluan of Table 5 lists all of the typical functions abstracted 
for the lltovature searched. 

Next, SDC exaiilnad each of the functlona Hated In colunn 1 of Table 5 to 
determine whether It vaa likely to be required when the IM eourae MnageMnt vlll 
be aided by a co«pute| ayatea — when the IM couraa will be part of the Advanced 
Inatructlonal Syatem |ms) . The Stateawnt of Work for AIS wae not avJllable when 
thla analyale wee perl|briied, ao aoM revtalon aay ba jrequlred at a later date. 

I 

\ 
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STEP 10: 
Prepare Instructiondl 
Management Plan 



I 



Substep 10-1 
Identify Management Functions 



Substep 10-2 
Detertnlne Functions Requiring 
Change Due to Individualization 



I 



Scbdtep 10-3 
Determine Functional Flow of 
Management Activities 



Substep 10-4 
Develop Plan for Each Function 
That is New or Requires Change 



I 



STEP 11 

Validate and Revise Materials 



Figure 30. Substeps in Preparing Instructional 
Management Plan 
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^The second colunn of Table 5 showa our declflons aa to the functlona that 
viXl probably be required to manage courees like the IH course vhen the AIS 
is operational* 

Ihlrd^ SDC decided which of the functlona In coluan 1» of Table 5 » would 
need to be performed for the IH courae when It la being managed In the manual 
mode. Here we made aome assumptlone about how students would be selected for 
the course* the extent to which the course might be revised due to changi^s In 
the job, and so on/ Column 3, In Table 5, shows the decisions made, A majoir 
decision waa to develop a "baro'^bones" management plan for the manual phaae. 
In this way» the manual management ayatem will not be overloaded with record 
keeping and data collection tasks that, while easy to perform in a computer^ 
aided systemi require a large ataff to perform manually. Therefore! functions, 
for which the column 3 declalon Is "desirable" rather than "required" are not 
Included in the manual ^^wageroent plan* 

SUBSTEP 10-2. DETERMINE FUMCTIOKS REQUIRING CHANGE DUE TO INDIVIDUALIZATION 

For this substep» SDC decided which IH functions for the manual mode (those 
In column 3 of Table 5 .) were new» that is not now being performedi or would 
require changes in procedure from the way they were now being performed* * 
Column % In Table 5 reflects these decisions t The final column in this 
table» column S» shows a Yes or No decision as to whether new planning is 
required for each function. Functions that are new or need changes have 
a Yes decision* Functions that can be performed eaaentially as they are 
performed now have a No decision^ as do functions not required* Functions 
with a Yes decision are thoae decisions to be described in detail in the 
manual Management Plan. C| 



ASSyMPTIOHS FOR THE MAKllAL MANAGEMENT PL^N 

Aasumptlona for the IM Course are brliefly described as followsi 



# No computer capability available for aiding in performance 
of management functions during the manual phase. 

• Functions not requiring change due to Individualization 

of the IM course will continue to be performed as they are 
performed nowi 

• Selection of students for the course will continue to be 
on the current basis* 

# Course objectives, task analysis » and instructional 
strategy will not require extensive revision due to 
Job changes during the manual phase* 

# Large-scale data collection and report preparation 
activities » so easily accomplished in a computer-aided 
system, are not appropriate for the IM fourse in the 
manual phase. These activities must be kept to the 
minimum required to assess student performance and 
provide basic course evaluation data during the manual phase 
to avoid staff overload* 

• Complex adaptive modelling and prescriptive algorithms are 
not appropriate to the manual phase either* These functions 
must be kept relatively simple until computer sfislstance Is 
available* 




88 



SUBSTEP 10-3. DETERMINE FUNCTIONAL FLOW OF MANAGEMENT ACTIVITIES 

In order to plan how IM management functions can be performed in the manual 
modet and to aid managers in organizing those actlvitiesy SDC prepared a 
functional flow chart of all the management functions » Including those not 
requiring changes. Column 3 of Table 5 was used as the baseline set of 
functions for this activity. Functions for which the notation "Required" 
appears are those included in the flow chart. Figure 31 shows>tha flow chart* 
the functions that are new or require change are outlined with heavy black 
boxes in Figure 31. Functions that can be performed essentially as they 
are performed now appear with thin black lines around the boxes. 

Next, SDC specified for each function the type of person most likely to 
perform it and the activities or tasks performed as part of the function* - 
Table 6 shows a part of this analysis^ for the studeHt placement , 
scheduling^ record keeping, and report preparation functions. Specifying the 
activities for each function enabled us to see where new management aids, such 
as logs, report forms, or decision-making rules, would be required, and to 
check to be certain that functions thought to require no changes could be 
performed without new aids or procedures* 
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SUBSTEP 10-4. DEVELOP PLAN FOR EACH FUNCTION THAT IS NEW OR REQUIRES CHANGES 

Column 5 of Table 5 determined which functions would require management 
planning. Each of these functions is described In the management plan, as 
they will be performed in the manual mode of operation. New report forms and 
other management aids are included in the descriptions. The manual management 
plan for the IM Course Is located In Appendix C. 

Some Comraents on Management Planning 

Some changes to an instructional system require policy decisions rather than 
pure technical decisions. Policy decisions must be made by school administra- 
tive personnel and other management personnel rather than by technical planners* 
An example of this kind of system change involves deciding how to schedule 
students who complete the course early, which Is jjery likely to occur In the 
IM course. Should these students be given leave until the normal time 
for Job assignment? Should they be used as course tutors for slower students? 
Technical planners can furnish information about the probable consequences of 
each alternative for the use of others In making this kind of decision, but 
the decision itself Is a matter of policy. 

One of the most Important areas to consider In management planning for a 
self-paced individualized instructional system is that of scheduling and 
allocating resources, and scheduling and locating students. Students will 
be completing self-paced lesson modules at different times and will require 
practical experience training on an irregular schedule. Students may not 
be physically located in a particular classroom all day. Unless a system 
is devised for organizing information about the availability of resources 
and the needs of students for these resources, a self spaced course will be 
too chaotic for the inatructor to offer help with course content when such 
help is needed. A log showing the location of lach student in the class is 
a necessity. A resource planning aid la also a necessity, enabling the 
Instructor to anticipate the neede of his students and make needed resources 



ERIC ^ 



4 



available to them. For most aelf-paced courses, money Is not available to 
provide each student with full-*tlme use of tape, slide, movie, or TV 
equipment, and Indeed the student does not require full-time use of this equip 
ment in the IM course. But, if the student cannot have access to such 
equipment when he needs it, the main advantage of self*-paclng Is lost, and 
students lose Incentive to proceed as quickly as they can through the course. 

A long delay in performance testing has a bad* effect also. Some forgetting 
is normal after learning. A delay In testing for some students and not for 
others puts the students suffering the delay at an unfair advantage. A dally 
scheduling log can be designed to overcome many of these problems. Ideally; 
It can be tried out on a dry-run basis and perfected before Its use In the 
classroom setting Is required. An Instructional management plan must address 
Itself to solving those kinds of problems. 
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STEP lis VA1.IDATE AND REVISE MATERIALS 

Once learning materials are developed, they must be tried out and then ' 
revised before they are used in training course^^. Almost always It Is 
found that aome Instructional frames and test Items are misunderstood by 
a significant portion of the group participating in the validation tryout 
and thus require revision. (Figure 32 presents Step 11 and Its substeps.) 

The validation and revision process is a cyclical one. When revisions 
have been made, substep 11-4, then a new validation plan Is developed to 
test the effectiveness of the revisions. 

For the IM course the validation was performed manually — without computer 
assistance. Where computer assistance Is available, as It will be In the 
completed AIS, validation data can be stored and processed using the com* 
puter and suitable programs. (see Appendix D for Validation Study & Data.) 

SUBSIEP 11- I. DEVKLOP VALIDATION PLAN 

Development of a plan for performing validation and revision is a crucial 
step in the validation process. A number of elements must be considered 
in the plan, each of whi^'a is discussed below. 

Purpo se of J^aljld a ti^i>n 

Two major purposes are possible for the kind of validation that is performed 
before materials are officially in use in a training course. These purposes 
are: 1) to improve instructional frames, test items, supporting materials 
and procedures so that they will be understood by most trainees and instruc- 
tional personnel, and 2) to assess trainee performance, learning rates, 
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STEP 11: 
Validate and 
Revise Materials 



Substep 11-*1 
Develop Validation Flan 



Subdtep 11-2 
Conduct Evaluation 



Subetep 11-3 
Determine Findings and 
Revisions Required 



Substep 11-4 
Make Revisions 



STEP 12 1 
Prepare 
Cost Analyses 
Report 



Figure 32. Substeps In Validating and Revising 
Materials 
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training times, efficiency of scheduling, and use of resources to make it 
possible to compare the effectiveness of the new materials with the old 
or with other sets of new materials and to aid In developing an effective 
management plan for the course. 

For the IM course validation, SDC chose to plan the validation for both 
purposes. Therefore, planning included specifying data to be gathered and 
evaluation measures to be used that could be utilized for improvement of 
the roaterlals and assessment of student performance and Instructional man- 
agement* 

Method of Conductinf^ Validation 

Several basic methods are possible for conducting a validacion. Ideally, 
Instructional materials would be validated using a large group of actual 
trainees In the course for which the new materials are being prepared, and 
the procedures, facilities, instructional personnel and so on would be 
those actually occurring in the course environment. Practically, It is 
difficult to gather all of these resources for the first validation to be 
performed. Also, trainees participating In such a validation study may 
not be adequately trained by the yet- to-be revised materials, which makes 
it difficult ^for training administrators to give their permission for such 
an experiment. Further, such a full-scale validation process is expensive 
and time consuming — usually too much so for totally untried materials. 
Therefore, initial validation trials are usually accomplished on a smaller 
scale. 

For the IM course, SDC planned a three stage validation process. These 
stages were as follows: 
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• S ta&e !♦ Individual assessment and critiques > \7hereln lesson tryouts 
and comments by non-Air Force personnel naive to the IM materials 
are used to make revisions prior to field trials. 

• ^-ta&e -2 . Small ^ roup field tryouts by IM studfencs, and critiques by . 
ATC specialists* Iten/frame analyses of student responses and comments 
«;ve performed. Performance scores and learning times are analyzed 

for each lesson element. Review comments from a group critique by 
Instructors and Job specialists are analyzed also. Analysis results 
Lire used to revise lesson materials, lnsttuctlon3> and test Items 
and to aid In firming up an instructional management plan, SDC used 
seven groups of four students each to try out each lesson In this 
phase* Within the Block I-III lessons, each of five groups tried 
one Lesson only. Within block IV, one group tried two lessons and 
the other group tried six lessons. 

• SAMg 3> Group field tests , supported by ATC instructors. Portions 
of the course that are to utilize the new materials are conducted 
much as if the new materials were officially in use. A major purpose 
of this stage is to verify the effects of Stage 2 revisions on student 
performance, learning rates, and attitudes. Another Important purpose 
l<i to test the operational integration oC course alements; In partic- 
ular, to identify problems and to shake down personnel requirements 
and prociidures, equipment, and training management functions. SDC 
used 20 students in this stage, each of which tried all block IV 
mfitec la Is, 



Va U^^ilon Measu r ^ J^M It hims, and Data Collection I ngtruments 

Three valUliU lou elements — validation measures, data items, and data collec- 
tion instriinients are all firmly linked together in a validation plan. The 
purposes of the validation determine to some extent the kinds f validation 
measures required. Once these measurer* are fully specified, the types of 
data to be collected are also determined. When types of data are determined, 
data collection instruments must be developed. These instruments must be 
appropriate for the kinds of data to be collected and the measure to be used, 
and also for the physical environment in which the validation will take place 
and for the resources and fuuds which caa be expended for valid.^tion purposes. 
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For Stage 1 of the IM course materials validation, SDC planned only an informal 
collection of data. No formal validation measures were specified and no data 
collection Instrutiieats developed. Validation team personnel distributed mate- 
rials to interested colleagues who volunteered to serve as ^'atudent8'^ Each 
'*student^' worked alone through the materiaL6> making notes as he progressed. 
Ad he completed the materials, he critiqued them for one or more members of 
the validation team. Validation team members then made revisions to materials 
on the basis of the critiques and on the performance of the ^'students*' on the 
individual items in the materials. 



For Stages 2 and 3 of the IM course materials validation, planning was more 
formal. The following types of data items were collected: 

1. Elar jed time spent on each lesson for each student. 

2. Eac; student's answers on enabling and criterion test Items. 

3. Nutober of wrong answers per student on each enabling and criterion 
test. 

A. Student atfcitudef> toward the materials and the individual mode of 
instruction. 

5. Critique comments from IM course instructors. 

6. Student background information, such as age, AQE score, and prior 
experience of training relevant to inventory management. 

7. End of Block test scores (Stage 3 only). 

All of these, except the last, serve a purpose in determining the kinds of 
revisions needed to materials and associated procedures. The student back- 
ground Information was collected for possible later use in relating student 
AFQT/AQE scores and experience to course time completion data or performance 
data. 



Several data collection forms were developed for use In the Stage 2 and 
Stage 3 validation trials. No special data collection form was required 
for recording each student's answers on enabling and criterion test items — 
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copies of the Instructional materials were used for this purpose. Elapsed 
time spent on each lesson eleir.ent (separate times for each element were 



collected for Stage 2 only) was recorded on a form like that ahovm In 



Figure 33, 
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Figure 33. Student Learning Times Recording Form 
(Stage 2) 



Figure 34 shows part of the attitude questloM.aita Uiat vas adninistered. 



ERIC 



99 



^) UNDE RSTANDABILITY OF MATERIAL 
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Figure 34. Portion of Attitude Questionnaire 
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A portion of the background data questiotinaire is shown in Figure 35. 
NAME_ ^ ' GRADE_ ^TIME IN GRADE 



SERVICE m. APQT SCORE^ 

AGE_ EDUCATION 



(GRADE COMPLETED OR DEGREE) 

AFSC 



AQE (Airman Qualifying Exam) Scores; 

Administrative 

General 

Electronic 

Mechanical 



Experience : 

1. Total years/months In military 



2. Total years/months In Supply Career Field 

3. Have you hads 

a. Supply Helper Experience 

(1) Months (2) AFSC^ 

(3) What did you do? 



Figure 35. Portion of the Background Data Queatlonnal 
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The following primary validation measures were planned for the Stage 2 
validation: 

li Me^n proportion of Items correct, per lesson^ 

2. Mean proportion of time saved (over existing lesson time) per lesson. 

3. Median attitude ratlngi for each lesson, for each attitude question, 
and over all lessons and questions. 

For Stage 3, the same validation measures were planned and several others 
were added: 

1. Performance Improvement, Stage 2 to Stage 3, per lesson and over 
all Block IV lessons. Differences In lesson mean scores were used 
to calculate these measures. 

2. Time saved improvement (over Stage 2 time saved) per lesson and 
over Block IV lessons. 

3. Number of students receiving remediation >er lesson. 

4. Mean End of Block test score. 

Alia l ysis Plans 

Data analysis plans are la/gely dependent on data to be collected and valida- 
tion measures selected. Dcita analysis forms should be developed during the 
validation planning so that analysis can proceed smoothly when data are avail- 
able. Forms must be carefully designed to minimize errors that can occur in 
combining data. If keypunching or optical scanning are going to be used to 
input data to a computer, forms should be specially designed for this purpose. 
SDC had a relatively small amount of data for the IM coursa validation and 
so it was decided that manual analysis, aided by electric calculators, would 

be most cost-ef f ecLive to reduce data for aiialysis. Analysis forms v/ere 
therefore designed for this use. Figure 36 shows Liie foru developed to 
summarize student performance data arid record results of measurement. 
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B-1 

STUDENT PERFORMANCE SCORES 
LESSONS 1-8, BLOCK IV 
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Figure 36, Sunmary of Student Performance Data Form 



ERIC 



103 



The forms shovm In Figure 37 were used to summarize student learning times 
and to calculate percent time saved over time in current classesi Times for 
current classes are shown In parenthesis above the summary fortn. 
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Figure 37. Summary Forms for Student Learning Times and Percent 
Time Saved 

Figure 38 shows now student attitude data were surmaxized* 
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Figure 38. Student Attitude Data Summary Foim 
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iA§a...^yAy*'.l?. i".^ Materials R evisions Plans 

A major purpose of the validation tryout is to provide data and Information 
to enable instructional oiateri^^^s developers to make appropriate revisions 
to the materials. Data and Infor^j^on useful for this purpose Include all 
of those discussed earlier in this Step, In addition^ student performance 
on each frame and test Item should be analyzed to determine the items on 
which students perform poorly, A standard or criterion must be set to define 
•'poor'^ performance. Generally, for self-instructional materials, a "good'' 
test item is one that is missed or answered Incorrectly by no more than 15% 
of the students. SDC could not use this criterion for the IM materials In 
the Stage 2 validation because only four students performed on any given item. 
Therefore, for the Stage 2! validation when an enabling or criterion test item 
was missed by one or more of the students or when the mean percentage of 
correct items for a lesson was less than 85%> then the lesson or teat item 
and Its associated Instructional frames were considered candidates for re- 
vision. For the Stage 3 validation, whui an enabling or criterion test item 
was missed by three or more students (85% of 20), or when the mean percentage 
of correct items for a lesson was less than 85%, the lesson or item and asso- 
ciated frames were considered candidates for revision. Example of the item 
analysis matrices are located In Appendix D, 

A standard should also be set to define **poor" learning times* In the military 
a general rule of thumb for this definition is that good self-paced individu- 
alized instructional materials should reduce learning time by at least 30% in 
coiapairison to the time allotted for the more traditional clussroom/laboratory 
method, SDC used this rule of thumb in setting a learning time standard for 
each lesson — mean time (over students) saved must equal at least 30% or the 
lesson would be a candidate for revision. 



A criterion or standard can be set vlth respect to attitude data too. SDC 
decided that for the Stages 2 and 3 validations, median attitude ratings for 
each leflson should be at least at the scale midpoint or the lesson would be 
a candidate for revision* 
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One of the most difficult aspects of the validation and revision process is 
to determine exactly what items should be revised — and in what way — to 
Improve performance. An analysis of the incorrect answers to test items 
provides valuable clues to Identifying problems within the materials. Analysis 
of student performance on instvuctional frames provides additional clues. 
GDC planned these kinds of analyses for the IM Materials. Figure 39 shows 
the data sheet that was used to record incorrect responses to test Items. 
Blanks in the form indicate that there were no Incorrect responses for a 
particular item. 



CRITKRION 



8 



LESSON 



IV 



BLOCK 



TEST ITEM ANALYSIS 





3 


6 


7 


8 


9 


WRONG 


X 


BKINKMAN 


B 




WRM 
ITEMS 


D ONLY 


8 ONLY 


4 


67 


CRAWFORD 


E 






C ONLY 




2 


83.5^^ 


DAY 












0 


100 


DILLON 


E 


B 




EC 


E 




67 
















- - ■■ ■ -1 



Figure 39. Item Analysis Recording Form. 
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SDC planned to revise Instructional ciaterldls on the basis of the Stage 2 
validation* The revised materials would then be used in the Stage 3 valida- 
tion and further revisions would then be made as requited by results of the 
Stage 3 validation. 

PlaoninA the Conduct of the Validation Trials 

The conduct of the validation trials must be carefully planned and coordinated 
vlth the administrative and Instructional staffs responsible for the course 
for which materials are being prepared^ The following factors must be con*- 
sidered; 

• Student flow and scheduling 

• Resources allocation and usage 

• Numbers and characteristics of students required 
e Constraints on tlme» funding* and logistics 

• Boles of valldstlon team members and instructional staff members 

• Administration and monitoring procedures 

For the IM course, SDC developed test conduct plans for the Stage 2 and 
Stage 3 validation trials. 

Stage 2 Validation Trials 

For the Stage 2 validation, small group field tryouts, SDC planned to administer 
the Block I**XII lessons to IM specialist course students at Lowrey AFB> four 
students per lessoni with one SDC observer loonltoring t^io students for each 
tryout* The block IV lessons were to be administered to two groups of four 
students t each student undertaking a sequence of lessons* All students were 
^to be selected by the Air Force from the ongoing course. One classroom was 
to be used for all sessions. Figure 40 shows the room configuration used. 
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• (1) 
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(2) 
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(1) 

(2) 
(3) 
(4) 



SDC observer 
Table 30*' x 45" 
Student seated 

Kodak Carousel slide projector 

HPI Telescreen "'T'^ Model HOST 
Rear projection screen 



(5) Wollensak Slide-Synch Audiovisual 
Cassette Playback Unit 

(6) Wollensak monophonic headphone ^ 
Model A-0483 

(7) Loaded slide tray and audio 
cassettes 

(8} Student instructions, PI itext, 
tests, etc^ * 



Figure 40* Classroom Configuration for Stage 2 Validation Trials 
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Plans for monitoring and administering the Stage 2 trials were as follows t 



e Brief students about illdatlon« 

# Administer I collect, and check background Information forms 

# Monitor instructional materials trial, recording beginning and ending 
clock time for each lesson element, answering student questions and 
recording student difficulties. 

# Obsetve student scoring of enabling tests* (Criterion tests were 
scored later by the validation team.) 

# Administer attitude questionnaire and the ATC Programmed Instruction 
Critique (Form 3ABR6A530-1). 

# Give instructional materials to ATC instructors and Job specialists 
for critique (materials to be reviewed and mailed back within two 
weeks) t 



Stage 3 Validation Trials 

For the Stage 3 validation trials SDC planned to administer the Block IV 
lessons to 20 IM specialist course students at T-owtey AFB, in conditions 
approximating those in which the lessons would actually be used. All students 
were to be selected by the Air Force from the ongoing course. Two classrooms 
were used for the sessions, one of which Wifl used only for the audiovisual 
equipment. Remedial instruction was giver, in both rooms. Figure 41 shows 
room configuration used. 

The validation team was to consist of two SDC project members and two ATC 
instructors. Plans for monitoring and administering the Stage 3 validation 
trials were as follows: 

• Brief students about the validation. 

s Administer, collect, and check background information forms* 

• Monitor instructional materials trial, recording beginning and ending 
time for each lesson, answering student questions and recording 
student difficulties. ^ 

s Score criterion test. 

0 Administer attitude questionnaire. 

^ Administer end of block test« 
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Figure 41 shows the toom configurations used. 

Classroom Configuration 



€) (51 IQ^ I©©1 iQQl 
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Audio Visual Room Table Layout 




X Validation Team Observer 

1 Instructor 

<P Table 

0 Student Seated 

(3) Kodak Carousel Slide Projector 

(S) UP! Telescreen *'T'\ Model #6057 
Rear projection screen 



O Wollensak Slide-Synch 
Audiovisual Cassette 
Playback Unit 

<S) Wollensak Monophonic 
Headphone J Model A-0483 

(2) Loaded Slide Tray and 
Audio Cassectes 



Figure 4L« Stage 3 Validation Trials Room Configuration 
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SUBSTBP 11-2. CONDUCT VALIDATION 

Once planning of a validation is completei the conduct of the trials is a 
relatively simple matter* The validation plan is followed unless unforeseen 
events make this impossible* For the small group trials , Stage 2, SDC was 
able to conduct the validation trials Just as planned. For the group field 
test validation* Stage 3| the Air Force requested SDC to limit the materials 
to be validated to those for Block IV » Lessons 1-8, with which request SDC 
complied. 

SUBSTBP 11-3. DETERMINE FINDINGS AND REVISIONS REQUIRED 

The parts of the validation plan pertaining to validation measures » analysis 
planning and revision planning are to be followed in performing this substep. 
It may happen that an analysis not anticipated at the planning stage will 
suggest Itself during the data analysis » and» 1£ appropriate data are avail- 
able » then such an analysis may be added during this substep provided enough 
funds and manpower are available to perform it. 

A summary of SDC's analysis findings and revision decisions is presented 
below. 

Stage One Findings 

Informal notes taken by the course developers during intensive debrieflngs 
with those who played the role of students In the Stage 1 individual assess- 
ments at SDC provided the basis for revisions made prior to field tryouts. 
In general » the print and audio-visual presentation technique proved to be 
satisfactory during this internal review^ though portions of the procedural 
instructions often require rewrite. No formal specifications or records 
were kept of the changes made. 
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&t%M Tvo Findings 

Four out of thirteen lessons had mean percent correct scores of less than 
85X> making those lessons candidates for revision. About a third of the 
total number of enabling and criterion Items were missed by at least one 
student, making those items and their associated instructional frames par- 
ticular candidates for revision. 

An analysis of the learning times for lesson elements versus times for the 
currently taught lessons shows that the mean time savings over all lessons 
was ranging from 34 to 70 percent per lesson and from 23 to 7A percent 

per student. 

Analysis of the attitude survey data shows that results generally support 
the positive findings with respect to performance and learning time. The 
median rating over all attitude items and lessons was 7 on a 10-point rating 
scale in which 10 was high or positive. Highest ratings were for items 
pertaining to whether the material holds students' interest and to the util- 
ity of the tape/slide presentations* 

Instructor critiques and the attitude data were used to guide the types of 
revisions made, as was the analysis of the incorrect answers for test items. 

Figure 42 shows the revision percentages for each lesson for the Stage 2 
validation* 
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Figure 42, Lesson Module Revision Percentages (Stage 2) 
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Stage Three Findings 

No Block IV .lessons had mean percent correct scores of less than 83% In the 
lerge group field trials* In the Stage 2 trials » four of these lessons had 
mean percent correct scores of 70^84 . SDC concludes that revisions made 
after the Stage 2 trials were effective in Improving student performance* 
In analyzing errors on particular items from the Stage 3 trials, it was 
found that about a fourth of the total number of enabling and criterion 
itema were missed by at least 15X of the students » making those items the 
focus of the Stage 3 revisions* 

An analysis of learning times for the new Block IV lessons versus times for 
the currently used lessons shows a mean time savings over all Block IV lessons 
of 47X'| with a range of 21 to 82 percent over lessons and a range of 34 to 52 
percent among students* 

Analysis of the ^tage 3 attitude data show very favorable attitudes towards 
the self^paced materials* The lowest mean ratings were still well above the 
midpoints on the rating scales i Lesson 2 received the lowest mean rating (7) 
and lesson 8 the highest (9)« 

Figure 43 shows the estimated revision percentages for each lesson for the 
Stage 3 validation* 
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to nearest 5%0 

Figure ^3. Stage 3 Validation Lesson Revision Percentages 
SUBSTEP 11-4 » MAKE REVISIONS 

Revisions to materials are made after each validation stage« SOC revised both 
, Instructional Items and procedures after the Stage 1 validation. The revised 
materials were then used In the Stage 2 validation and revised materials 
resulting from the Stage 2 validation were used In the Stage 3 validation. 
This cyclical validation process will be continued by the Air Force after the 
materials arr in use, in the IM course. 

The revision process itself is like that for the original preparation of the 
Instructional materials* Step 9* 

STEP 12. PREPARE COST ANALYSIS REPORT 

A cost an«lysl8 is not always a mandatory step in the preparation of instruc- 
tional materials. Its preparation is to be encouraged however » since valuable 
information that can be used in the future to iiiiprove instructional materials 
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STEP 12 
Prepare Cost Analysis 

ZIXZ 

Substep 12-1 
Prepare Records of 
Direct Man-Hour 
Data 

Substep 12-2 
Prepare Records of 
Materials Cost 

Substep 12-3 
Prepare Production 
Configuration Cost 
Estimate Rationale 

L_ 

Substep 12-4 
Prepare Production 
Configuration Cost 
Estimate 

End 



Figure 44, Substeps In Preparing Cost Analysis 
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development procedures is therctby preserved and made accessible. There is 
a certain amount of controversy at present about the amount of time that 
Is required to develop effective programed instructional materials. If 
all teams developing such materials would keep manpower tiroes and material 
costs 9 much more accurate estimates would soon be available. 

Figure ^4 shows the substeps involved in preparing a cost analysis report 
of the sort requested by the Air Force for the IM course. 

SUBSTEP 12-1. PREPARE RECORDS OF DIRECT MAN-HOUR DATA 

As an instructional materials development project begins* forms should be 
developed to record man-hours expended* The form used by SDC for the IM 
course is shown in Figure 45. 



I — 



Man-Hour Recording Form 



Direct Man 
Hours 



Phase I 



Analysis of IM Course 
Selection of Course Segments 
Preparation of Technical Report 



104 



37 



175 



Phase II 



Task Analysis 

Development of Learning Objectives 



417 



332 



Figure 45. Portion of Man-Hours Data Collection Form 
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SUBSTEP 12-2. PREPARE RECORDS OF MATERIALS AND RELATED COSTS 

A form similar to the Man-Hour Recording form was used to record costs of 



tndterlaldi ttaveli dupllcatloni and 


so forthi Figure AG 


shows a 'Dortion 


of the form used for this purpose* 






Materials and 


Related Costs Form 


— 1 






Costs In 
Dollars 


Phase IV 






Editorial support 




729 


Reproduction 




2,874 


TDY Travel 




.324 


* Black Slide Trays (180) 




2,614 


35 MM duplications of 545 (29 


sets) 


4,794 


Visual arts support 




340 


Tape cassette duplication of 


11 (29 sets) 


1.058 


. « 




12,733 



Figure 46. Portion of Materials and Related Costs Form 



SUBSTEP 12-3. PREPARE ESTIMATE OF PRODUCTION CONFIGURATION COSTS RATIONALE 

SDC was asked what materials development costs would be if Instructional 
development work was performed in a production configuration* This estimate 
was to be based on the costs we incurred in a configuration that was assumed 
to be basically research and development oriented. 
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A rationale to account for the differences In coats in the two types of 
configure tlona is therefore required* SDC preparod two sets of assumptions 
£or this substep to account for cost differences due to configuration* These 
eets of assumptions appear in Part Three, Phase IV of this report. The first 
•et of assumptions applies to costs for a true R&D configuration versus a 
production configuration. The second set applies to costs ixi the SDC con- 
figuration, which was considerably more production-oriented than would be 
many research and development efforts, versus reproduction configuration. 
In a true RiD configuration, the methodology and strategics required to 
produce the instructional materials would have been developed especially 
for the IM project, adding considerably to direct man-hour costs. In SDC's 
configuration, methodologies and strategies for performing each phase of 
the work had already been developed for earlier projects and needed only to 
be adapted to the IM project. In a production configuration, existing 
methodologies are adapted to a number of different projects at the same 
time, so that personnel and materials are shared over a number of similar 
projects and the costs of special production equipment are shared also 
among a number of projects. Persons performing a specific task for an 
instructional development project tend to be quite specialized, in compar- 
ison to RiD project personnel, because the volume of work for that specific 
task is great. Therefore, specialists at different salary and skill levels, 
and with varying kinds of skills, can be utilized efficiently, lowering 
direct man-hour costs. For example, in an R&D configuration, one person 
might develop instructional strategies, perform task analyses, write in- 
structional frames, and record audio material. In a production configur- 
ation, these tasks would be assigned to different people, at widely dif- 
fering salary levels. 

In a military setting, any of the three configurations described can be 
found, and each would be appropriate for a different set of conditions. 
The production configuration Is, of course, to be preferred when conditions 
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make tha appllcationa possible. And» even then^ an R&0|Unlt is required 
to serve the production uniti so that advances in methodology, equipment, 
or theory can be made and applied to production. 

Conditions that make a production configuration possible are as follows: 



• Organization Is large enough to allow great specialization among 
workers and a variety of equipment In-house. 

• Volume of work is great enough* that each specialist group can be 
kept busy full timet 

• Instructional materials development methodologies , strategies i and 
techniques are efficient enough and versatile enough to be adapted 
to many different kinds of training needs. 

• Advances in the state-of-the-art are infrequent enough that pro- 

' duction personnel do not have to be retrained for each project or 
at short intervals. 

• Well-trained personnel for each specialty are available to the 
organization. 

As^you may. h§ve guessed,^ there are few true production cor*=tgurations in 
existence today. The state-of-the-art Is const;intly changing. Methodologies 
are constantly being improved, or at least changed. It lo difficult to find 
well- trained programmed instruction frame writers ~ indeed this task is still 
considered to be an art or a special talent by many persons working in the 
field. However, the military has a unique opportunity in the area of instruc- 
tional materials development, due to its size, its enthusiasm for training 
or being trained, and its willingness to undertake large-scale research 
projects. 

SUBSTEP 12-4. PREPARE PRODUCTION CONFIGURATION COST ESTIMATE 

Cost figures and the production estimation rationales were used by SDC to 
prepare the cost analysis. This analysis is contained in Appendix E. It 
reports cost figures developed in substeps 12-1 and 12-2, lists 
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the estimation assumptions from substep 12-3 » and contains estimates of the 
amount of reduction in costs that might be expected in a true production 
conf iguratlon« These reductions are relatively small due to the fact that 
SDC^s configuration is production-^oriented to a considerable degreei Figure 
47 shows a portion of the summary of these estimates. 

Other types of cost analyses might be useful in the military setting* For 
example » a cost-benefit analysis might provide useful information in which 
costs Incurred in development of materials are evaluated with respect to 
improvements in trainee performance or to decrease In aptitude levels re*- 
qulred to complete the course satisfactorily* 



P otential Cost Reductions 
In a Production Configuration 



Phase III 



% Reduction 



Develop Instructional Materials 

Perform Pilot Study 
« 

Develop Management Strategy 
Prepare Technical Report 



5% 



5% 



0% 



20% 



Figure 47. Part of the Summary of Potential Cost Reductions 
Estimated from SDC's Costs 
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APPENDIX A 
CENTRAL FILE LISTING 



INVENTORY MANACSMENT SPECIALIST CENTRAL FTI.K T.TSTTMfi 



001 
002 

003 
004 
005 
006 
007 
008 
009 
,010 
Oil 
01^. 
013 

014 
015 
016 
017 
018 
019 
020 
021 
022 
023 
024 
025 



DOCUMENT 
CONTROL i! 

64530-1 
STS 645X0 

TDLO-1 

64530-1-1-2 

64530-1-1-3 

64530-1-1-4 

64530-1-1-5 

64530-1-1-6 

64530-1-T-7 

64530-1-1-8 

64530-1-1-9 

64530-1-1-10 

64530-l-,I-ll 



64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 



1-12 

I- 13 

II- l 
II-2 
II-3 
II-4 
II-5 
II-6 
11-7 
II-8 
II-9 
11-10 



SUBJECT 

PLAN OF INSTRUCTION - Inventory Management Specialist. 

Inventory Management Specialist and 

Inventory Management Supervisor, 7 November 1969 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 



Lesson Plan 
Stock Llsc 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lesson Plan 

Lasson Plan 
Functions 



Orientation 

Airtnan Supply Career Field 
Coiomunlcations Security 
USAF Supply System 
Air force Stock Fund 
USAF Supply Manual 
Master Cross Referenc(|^Llst 
Illustrated Parts Breakdown 
Cataloging Handbooks 
USAF Federal Supply Catalogs 
USAF Interchangeable and Substitute Grouping 

Index of Cataloging PubllcatTyns 
Research Functions 
Standard Base Supply System 
Internal and External Records 
File Maintenance Procedures 
Remote Operations 
Supplies Issue Procedures 
Repair Cycle and DIFM Control 
Rejecte and Inquiries 
Bench Stock Operations 
Supplies Inventory 

Item Accounting & Mainf:endnce Support 



026 64530-1-III-1 



Lesson Plan ^ Stock Control Levels 
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DOCUMENT 
ntt » CONTROL » 

027 64530-1-111-2 
02$ . 64530-1-1 n-3 

029 64j^30-l-III-4 

030 64530-1-III-5 

031 64530-1-III-6 

032 64530-1-III-7 

033 64530-1-III-8 

034 64530-1-III-9 

035 64530-i-ni-lO 

036 64530-1-III-11 

037 64530-1-III-12 

038 64530-1-111-13 

039 64530-1-111-14 

040 64530-1-III-15 

041 64530-1- IV-1 

042 64530-1-IV-2 

043 64530-1-IV-3 

044 64530-1-1V-4 

045 64530-1-IV-5 

046 64530-1-IV-6 

047 6*530-1-17-7 

048 64530-1-IV-8 

049 64530-1-1V-9 

050 64530-1-IV-lO 

051 64530-1-lV-ll 

052 64530-1-XV-12 

053 64530-1-IV-13 

054 64530-1-1-2 (N) 

055 64530-1-1-3 (N) 

056 64530-1-1-4 (N) 



SUBJECT 



Lessor Plan 


m 


Initial Spares Support Lists 


lafson Plan 


m 


MILSTRIP RequUltions 


Lasion Plan 


m 


MILSTRIP Related Products 


Le&aon Plan 




Base Procurement 


Lesson Plan -* 
Reconciliation 


Due-Out Revalidation and Hue-'In 

* 


Lessen Plan 




Materiel Control 


Lesson Plan 




Monitoring Priority Requirements 


Lesson Plan 




Receipts & Due-^Out Releases 


Lesson Plan 




Stock Balance & Consumption Reports 


Lesson ?lsn 




Base Excesses 


Lesoon Plsn 




Shipments & Transfers ^ 


Leason Plan 




Document Control 


Lesson Plan 


mm 


Stock Control Functlona 


Lesson Plan 




Manual Accounting Systems 


Lesson Pl«n 


- 


Air Force Equipment Management System 


Lesson JPlan 


— 


Allowarice documents ^ 


Lesson Plan 




Authorisation Proo^dures 


Lesson Plan 




EAIO Records 


Lesson Plan 




Equipment Issue Procedures 


Lesson Plan 




Equipment Turn**In Procedures 


Lesson Plan 




Adjusting EAIO Records 


Lesson Plan 




Inter*-<]ustody Transfers 


Lesson Plan 




RQ( Procedures 


Lesson Plan 




Reports and Special Procedures 


Lesson Plan 




Equipment Inventories 


Lesson Plan 
Procedures 




Individual Equipment & Tool Issue 


Lesson Plan 




Equlpfment Management Functions 



Study Guide/Workbook - Airman Supply Career Field 
Study Guide/Workbook - Coomunl cat Ions Security 
StU'iy Gulde/tforkbook - USAP Supply System 
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FILE I 



DOCUMENT 
CONTROL I 



057 64530-1-1-5 (N) 

058 64530-1-1-6 (N) 

059 64530-1-1-7 (N) 

060 64530-l-I-8(N) 

061 64530-l-I-9(N) 

062 64530-l-I-10(N) 

063 64530-1-1-11 (X) 

064 64530-1-1-11 (X) 



065 
066 
067 
068 
069 
070 
071 
072 
073 
074 
075 
076 
077 
078 
079 
080 
081 
082 
083 



64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
645S0-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 
64530-1- 



I- 12 (N) 
•II-l 

II- 2 
■II-3 
■II-4 
11-5 
11-5 
II-6 
II-7 
11-8 
1 1-9 

II- 9 

•III-l(N) 

III- IA 
•III-IB 
III-2(N) 
•III-3 (N) 
III-4(N) 
-III-5(N) 



084 64530-l-III-6(N) 



SUBJECT 

Study Guide/Workbook - Air Force Stock Fund 

Study Guide/Workbook - USAF Supply Manual 

Study Guide/Workbook USAF Master Cross Reference List 

Study Guide/VJorkbook - Illustrated" Parts Breakdown 

Study Guide/Workbook - Cataloging Handbooks 

Scady Guide/Workbook - USAF Federal Supply Catalogs 

Workbook - USAF Interchangeable & Substitute Grouping 
Stock Lists 

Programmed Text - USAF Interchangeable & Substitute 
Grouping Stock Lists 

Study Guide/Workbook - Index of Cataloging Publications 

Study Guide/Workbook - Standard Base Supply System 

Study Guide/Workbook - Internal and External Records 

Study Guide/Workbook - File Maintenance Procedures 

Study Guide/Workbook - Remote Operations 

Workbook - Supplies Issue Procedures 

Progratnmod Text - Supplies Issue Procedures 

Study Guide/Workbook - Repair Cycle and DIFM"Control 

Study Guide/Workbook - Rejects and Inquiries 

Study Guide/Workbook - Bench Stock Operations 

Workbook - Supplies Inventory 

Progiammed Text - Supplies Inventory 

Study Guide - Stock Control Levels 

Workbook - Stock Control Levels 

Workbook - Stock Control Levels 

Study Guide/Workbook - Initial Spares Support Lists 

Study Guide/Workbook - MILSTRIP Requisitions 

Study Guide/Workbook - MILSTRIP Related Products 

Study Guide/Workbook - Inventory Management Specialist 
Course 

Study Guide/Workbook - Due-Out Revalidation and Due-In 
Reconciliation 



085 64 530-1- I I 1-7 (N) Study Guide/Workbook - Materiel Control 
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DOCUMENT 



FILE 1 


CONTROL # 


SUBJECT 


086 


64530-l-III-8(N) 


Study Guide/Workbook - Monitoring Priority Requirements 


087 


64530-l-III-9(N) 


Study Guide/Workbook - Receipts and Due-Out Releases 


088 


64530- I •III-IO(N) 


Study Guide/Workbook^ Stock Balance and ConaumDtion 
Reports 


089 


64530-l-III-ll(N) 


Study Guide/Workbook - Base Excesses 


090 


64530-l-III-12(N) 


Study Guide/Workbook - Shipments & Transfers 


091, 


64530-l-III-13(N) 


Study Guide/Workbook - Document Control 


092 


64530-1-III-14 (N) 


Workbook - Stock Control Functions 


093 


64530-l-III-16(X) 


Study Guide - Manual Accounting System 


094 


64530-1- IV-l(N) 


Study Gulde/Workbo6k 


095 


64530-1- IV-2(N) 


Study Gulde/Wotkbook - Allowance Documents 


096 


64530-l-IV-3(N) 


Study Guide/Workbook - Authorization Procedures 


097 


64530-1-IV-4 (N) 


Study Guide/Workbook - EAID Records 


098 


64530-l-IV-5(N) 


Study Guide/Workbook - Equipment Issue Procedures 


099 


64530-l-IV-6(N) 


Study Guide/Workbook - Equipment Turn-In Procedures 


100 


64530-l-IV-7(N) 


Study Guide/Workbook - Adjusting e!aiD Records 


101 


64530- 1-IV-8(N) 


Study Guide/Workbook - Intercustody Transfers 


♦ 102 


64530-l-IV-9(N) 


Study Guide/Workbook - REM Procedures 


103 


64530-1-IV-10(X) 


Workbook - Reports and Special Procedures 


104 


64530-l-IV-10(E) 


Proftraoaned Text - ReDorts and Sneclal'ProcpdurPfl 


105 


64530-l-IV-ll(N) 


study Guide/Workbook - Equipment Inventories 


106 


64530-l-IV-12(X) 


Workbook - Individual Equipment and Tool Issue Procedures 


107 


64530-l-IV-12(X) 


T 

Programmed Text ^ Individual Eauloment And Tool Ifiaufi 
Procedures 


108 


64530-l-IV-13(N) 


Workbook - Equipment Management Functions 


109 


HO TSLS-2 


Federal Supply Catalog - Mister Cross Reference List 


110 


HO TSLS-3 


Master Index 


llOA 


TSLS-4 


AFM 67-1 Volume I - Part One 


111 


HO TSLS-5 


AFM 67^1 Volume II • part Two 


112 


©'♦SJO-l-I-GEN-l (N; 


Handout - General and Research 


112A 


64530-1-1 


Handout TDLO-1 Orientation 


113 


64530-1-1-2 (X) 


Handout - Airman Supply Career Field 


114 


64530-1-1-3 (X) 


Handout - Communtcatlorts Security 
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FILE # 



DOCUMENT 
CONTROL # 



SUBJECT 



115 


64530-1-1-4 (X) 


Hi^ndout 


- USAF Supply System 


116' 


64 530-1- 1-5 (X) 


Handout 


- Air Force Stock Fund 


117 


64530-1-1-6 (X). 


Handout 


- USAF Supply Manual 


118 


64530-1-1-7 (X) 


Handout 


- USAF Master Cross Reference List 


119 


64530-l-I-8(X) 


Handout 


- Illustrated Parts Breakdown 


IIVA 


0^530-1-1-0 


Handout 


- General Airplane (T.O. IT-38A-4-1) 




04>JU-JL— i-l^ 


Handput 


- Cataloging Handbooks 


121 


64530-1-1-9-1 


Handout - Cataloging Handbook (H2-l)i Part 1, Groups 
and Classes 


122 


OHOJO-l-I-^f-Z 


Handout 
Index 


- Cataloging Handbook (H2-3), Part 3, Alphabetic 


too 


^/.C^A i T iA/V\ 

oh530-1-I-10(X) 


Handout 


- USAF Federal Supply Catalogs 


124 


64530-1-1-10-1 


Handout - Federal Supply Catalog, Identification List 
(FSC 1610) 


125 


64530-1-1-10-2 


Handout 


- Introduction to Federal Supply Catalog (C-l-AF) 


126 


64530-1-1-11-1 


Handout - USAF Interchangeability and Substitution 
Grouping Stock List 


127 


64530-1-1-11-2 


Handout 
System 


- USAF Stock List and Related Publications 
(S-1) - . . 


128 


04530-1-1-12-1 


Handout - Index of USAF and DOD Federal Supply Catalogs 
and R3lated Cataloging Publications USAF S-2A-1 


129 


64530-1-1-12 (X) 


Handout 


- Index of Cataloging Publications 


130 


64530-1-I-13CN) 


Workbook - Research Functions 


131 


64530-1-1-13- 
A&B(X) 


Handout 


- Research Problems 


132 


64530-1-II-1 


Handout 


- Standard Base Supply System 


132A 


64530-GEN-IlCN) 


Haydout 


- Codes, Data Elements, and Formats 


133 


64530-1-II-2 


Handout 


- loternal and External Records 


134 


64530-l-II«3 


Handout 


- File Maintenance Procedures 


135 


64530-l-IT-A 


Handout 


- Remote Operations 


136 


64530-1-II-6 


Handout 


- Repair Cycl^j and DIFM Control 


137 


64530-1-II-7 


Handout 


- Rejects and Inquiries 


138 


64530-1-II-8 


Handout 


- Bench Stock Operations 


139 


64530-1-11-10 


Handout - Item Accounting and Maintenance Support 
Functions 
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BEST copy AVAILABLE 



DOCUHENT 



FILB i 


CONTROL 1 






SUBJECT 




64530'1-GEN-II(N) 


. Handout 




Coded 9 Data Elemental and Formats 


141 


645?0-1.'OEN-III(N) Handout 


- 


Codes 1 Data Blementa, and Formats 




*4530-l-lII-2(X) 


Handout 


M 


Initial Spares Support Usts 


143 


645:i0-i 'Ill-aCX) 


Handout 


m 


MILSTRIP Requisitions 


144 


645^0 •V"lll-4(x) 






MILSTRIP Related Products 


U5 


64530-1 III-5(X) 




- 


Base Procurement 


146 


645J0 t III-6<X) 


a ici 1 1 vj v vA i> 


- 


Due-Out Revalidation and Due-In Reconclllatlo 


147 


645:iC-i"IJI-7(X) 






Materiel Control 


148 


645"»0 -1II-8(X) 


HAndoiit 




Monitoring Priority Requirements 


149 


6453' l-IJI-9(X) 


Handoufc 




Receipts and Due-Out Raleases 


150 


645'^0-l-III-10(X) 






Stock 'Balance and Consu^uption Reports 


151 




nsnouu L 




Basa Excesses 


152 


64530- 1-T.1I-12(X) 


Handout 




Shipments and Transfers 


153 


64530-A-in-13(X) 


Handout 


- 


Document' Control 


154 


64530-i in-14(X) 


Handout 




Stock Control Functions 


155 


64530- I -GEN- IV (N) 


Handout 




Codes J Data Elements ) a\id Formats 


. 156 


64530'.V IV~1(X) 


Hando^.t 




Air Fort e Equipment Mamigeinent 


157 


64530-l-IV-2(X) 


Handout 




Allo/anf a Documents 


158 


64530-J-tV-2-l 


Handout 




APR 0-10 Index of Allowance Source Codes 



159 

160 

161 
162 
163 
164 
165 
166 
167 
166 
169 



64530 -1-17-2-2 



64530-1 IV-^ 



64530-1 
64530-1' 
64530-1' 
64530-1 
64530-1- 
64530*1 
64530-^i- 
64530-1 
64530-1 



■1V-3(X) 

-IV-4(X) 

•IV-5(X) 

■IV-6(X) 

'IV-7 (X) 

-IV"8(X) 

>1V-9(X) 

-lV-ll(X) 

-:IV^-13"(X) 



and Organization Tables 

Handout - TA006 - Organizational and Administrative 
Equipment 

Hand'^'jt - TA016 - Table of Allowance for Personal and 
Special Purpose Clothing and Equipment USAF 

Handout - Authoi^zation Procedures 

Handout - EAID Records 

Handout - EqulpiPent Issue Procedures 

Handout - Equipment Turn-in Procedures 

Handout - Adjusting EAID Records 

Handout - Intcrcufitody Transfers 

Handout - Rai Procedures 

Handout - Squi ;>ioeiit Inventories 

Hax\dout - ?.quljpO!eT:i£ Management Functions 
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DOCUMENT 
FILE If CONTROL 0 

170 64530-1-1-13 

171 64530-1-II-3 

172 64530-1-II-5 

173 64530-1-11-5 

174 64530-1-II-5 

175 64530-1-II-6 

176 64530-1-1 1-7 

177 64530-1-II-8 

178 64530-1-II-10 

179 64530-1-III-5 

180 64530-1-III-7 

181 64530-1-III-8 

182 64530-1-III-12 

183 64530-1-III-14 

184 64530-1-IV-lO 
J.85 64530-1-IV-12 

186 64530-1-17-1*3 

187 RFQF33615-71-Q- 
2181 

188 

189 64530-1 

190 "64530-11 

191 64530-111 

192 64530-IV 

193 AFM 67-1 

194 AFM 67-1 

195 , AFM 67-1 

196 AFM 67-1 
196a AFM 67-1 
196b AFM 67-1 

197 AFM 67-1 
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Chart Summary of PubllcationG used In Block I 



Handout 


- Problem 12 




Handout 


- Problem 1 and Problem 2 




Script 


- Supply Issue Procedures 




Problem 


1, Problem, 2, and Problem 3 




Handout 


- Problem 9 DIFM Listing 




Handout 


- Problem 3» Problem 4, Problem 5, Problem 


6 * 


Handout 


- Problems 10, 11, 12, 13 and 14 




Handout 


- Problems 21, 22, 23, 24, 25, 26, 27, 28, 


29, 


30, 31, 


32, 33, 34, 35, 36 


Handout 


- Problem^ 12, 13, 14, 15 




Handout 


- Problems 4 and 5 




Handout 


- Problems 4 and 5 




Handout 


- Problems 7, 8 and 9 




Handout 


- Problems 23 thru 37 




Handout 


- Register of Control Numbers - Problems 1 


and 2 ' 


Handout 


- Problems 1 thru 12 




Handout 


- Individual Evaluation Project - Problems 


22~37 



Development of a Prototype of an Individual Instructional 
System (Inventory Management Specialist Course) 

Occupational Survey Report for the IM Specialist 

Written and Performance Test (A-F) 

Written and Performance Test (A-P) 

Written and Performance Test (A-F) 

Written and Performance Test (A-F) 

Vol. I, Part One, Chapters 2, 10, 12 

Vol. I, Part Four, Chapters 3, 12 

Vol. II, Part One, Chapters 1,3,4,6,7,8,9,11,14,16,17 

Vol. II, Part Two, Chapters I thru 23 

Vol. II, Part Two, Amendments 7,8,9 

Vol. II, Part Two, Amendments 11, 12 

Vol. IV, Part One, Chapters 5,6,7,8,9,10,19,21,22 

1 30 



4 



DOCUMENT 

FILE 9 CONTROL S UBJECT 

198 CDO 6A533 Inventory Management Specialist) Volumes 2, A» 5 

199 CDC 6A533 Workbook, Volumes 1, 2, 3, A 

200 CDC 64533 ^ Handout (TSLS-4) AFM, Vol. I - Part One 

201 Block I Transparenclas 117-123 (Missing) 

202 Block II Transparencies 222, 224, 225, 226, 227, 231-234 

203 Block III Transparencies 315 

204 Block IV Transparencies 40-1434 

205 AMRL-TR-6851 LEARNER-CENTERED INSTRUCTION (LCI): Volume II ~ Job 

Behavioral Description for AFSC 322X1R, August 1968 

206 AFSC 645X0/647X0 UNITED STATES AIR FORCE JOB INVENTORY, Inventory 

Management and Materiel Facilities Career Ladders, 
15 July 1971 
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APPENDIX C 



MANAGEMENT PLAN 
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PESCRimON or CHANGES REQUIRED FOR m SELECTION MANAGEKBW 

A plan for each new functldn or function for which changes are required due 
to the effect of IK course individualization is offered In this section* i 
For the eake of clarltyi functions that must be performed /but for which no 
changes are required^ are listed with the notation No 0iahgea Required/ 

Collect Student Background and Characteristics Data > 
No changes required* 

Estimate Student Course Completion Date 

The purpose of estimating of each student's completion date is three^folds 
to aid in student incentive management; to aid In predicting effective 
allocation of instructional resourceu; and to assist ih scheduling students 
for assignment to non^-lndivlduallzed portions of the course and for job 
assignment after course completion* 

the Initial estimate of completion dates will be based on student background 
data» such as AFQT 8Core» AQE score for A and G» and student experience in 
the career ladder prior to the course. Until more empirical data are collected 
on the rates of student progress, it Is difficult to provide an adequate 
algorithm for estimation of completion dates. . It is suggested that the 
time values shown in Table C-1 be assigned for use until actual data are 
available* Validation studies completed show generally greater tit e "Savings 
than those shown In Table C-1* 



Table C-li Initial Estimates of Course Completion Dates 



Student Background and 
Characteristics Category 


Estimated Time to Complete 
individualized Portions of the Course 


AFQT Category 1 or 2; AQE A and/ot G 
70 up; has been a Supply Helper or 
Apprentice Specialist prior to 
course* 


50% of curirerit coiirAA t:ime« 


AFQT Category 1 or 2; AQE A and/or G 
60**69; has been a Supply Helper or 
Apprentice Specialist prior to course. 


55% of current course time* 


AFQT Category 1 or 2; AQE A and/or G 
60 up; has not been a Supply Helper 
or Apprentice Specialist* 


60% of current course time* 


AFQT Category 3; AQE A and/or G 

60 up; has not been a Supply Helper 

or Apprentice Specialist* 


> 

65% of current course time* 


AFQT Category 3; AQE A and/or G 

60 up; has not been a Supply Helper 

or Apprentice Specialist 


70% of currept course time* 


AFQT Category 1-4 » AQE A and /or G 
SO-60; has been a Supply Helper or 
Apprentice Specialist 


75% of current course time* 


AFQT Category 1-4, AQE A and/or G 
50**60; has not been a Supply Helper 
or Apprentice Specialist* 


80% of current course time* 


AFQT Category 1-4; AQE less than SO* 


90% of current course time* 
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theae initial completion estimates should become a part of the initial Student 
Claiming Report and thus should be supplied by the administrative atafl! to 
InatructprSf An. instructor can then discuss this date with the student and 
aicfdain the process whereby the date will be revised to correspond wl(h student 
progress. 

Prepare Reports for Student Planning and Scheduling 

Because individualised portions of the IM course will be self^pacedt instructors 
will' need aids for organising and structuring their planning activities. A 
report form in which the progress of an individual student can be suninarized 
will enable th6 instructor to evaluate the student^s progress and to counsel 
hla more effectively. Figure c-1 shows a report form that is useful in 
planning and fot analysis within individualized instructional systems. Ideally^ 
the Student Report would be updated daily ■ 

ALLOCATE AMP SCHEDULE RESOURCES AND STUDENT S 

Ins truc>;ional resources (facilities^ equipment » materials » instructors^ etc.) 
Buvt be initially allocated by the administrative staff and then) scheduled on 
a daily basis by course instructors. Individualization of the course increases 
allocation and scheduling problems somewhat » compared to the traditional Instruc*- 
tional activities as a group. The fact that only a portion of the course is 
Individualized could Increase scheduling problems al80» particularly the schedr 
uling of students. 

Several routes through the IM course may be feasible^ but the simplest and 
eaaiest to schedule would be to group'-pace the lesson modules In Blocks I-III 
and reserve self-*pacin^ for Block IV. 



STUDENT REPORT FORM 



Name 

SSN 

Class 



AQE A \ G 
AFQT 



Start Date 



Completion Date 



Pre-^Course Experience 



Estimated completion dates for Lesson Modules l«»8s 
1 2 3 4 5 6 7 8 



Module £ntry/Completioa Data and Criterion Task Scores; 
(S) Enter date and tine 

(D Enter S (Satisfactory) or U (Unsatisfactory) 



Date 



TllDC 



Module 1 begimt 
Module 1 comp • : 
Task 1: 
Task 2i 
Task 3i 
Task A: 

Module 2 begun: 
Module 2 comp* i 
Task i: 
Task 2: 



Module 
Module 



Module 
Module 



3 begun: 
3 ot>mpt : 
Task 1: 
Task 2: 



begun: 
cotnp • : 
Task 1: 
Task 2: 



(D 



Module 5 begun: 
Module 5 cotnp » : 
Task 1 : 

Module 6 begun: 
Module 6 comp. : 
Task 1: 
' Task 2: 
Task 3: 
Task A: 

Module f 7 begun: 
Module 7 comp« t 
Task 1: 
Task 2: 
Task 3: 
Task 4: 

Module 8 begun: 
Module 8 comp. : 
Task 1: 
Task 2: 
Task 3: 



Date Time 



Progress Summary: 
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' the riaj t diaadvahtAge Ij that for the first 30 hours that have been 
jLhdivU^ tha benefit's of self'-paclng are lost. Students complete arv 

Injdlvldu^l self-paced ittodule and then return to group-paced Instruction. 
Students who finish before the time allotted for the lndlvlduali«e«J Inistructiort 
Is up will not be a problem particularly, since they can. be dismissed fbr the 
day or can assist instructors as tutors and demonstrators. Exceptionally 
slow students will be somewhat more of a problem in that they may not be able 
to finish In the allotted time. A solution to this problem — washback, 
extra time » remedial Instruction or another action — will be best arrived 
at as a policy decision of scliool administrators, taking into account 'the fact 
that practice over several hours is not usually an effective way to produce 
learning, particularly in slow students. 1 

' A possible alternative to group-pacing the first three blocks of instruction 
would be to take the individualized lessons out of their current sequence among 
group-paced lessons and group them all together. Subject matter experts 
would have to be consulted to identify any learning problems likely to be caused 
by altering the normal sequence of course content, (g,^ 

•Assuming that Block IV will be individualized and self-paced, to avoid 
scheduling problems it will be most satisfactory to utilize several of the 
instructors to work within this last portion only, as is the practice in the 
current course. This set of instructors ca« accept new Block IV students each 
week to replace students who have completed the previous course modules. Each 
instructor will be responsible for one or two lesson modules only. This 
allocation method allows the school to keep the instructor-Student ratio 
balanced more efficiently than would be possible if one Instructor Rt^yed with 
a particular group of students until the last one finished the coutsa* 
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Table C-2 shows the planned student locations within Block IV and gives 
tequlreoientd for nunibets of instructors, carrels» and sets of audiovisual 
ttatetlalS) rooms, and nets of progratnmed instructional aaterials. 

Table IM Course Resource Requirements for Block IV 

Carrels and 





Room 


Materials 


Instructors 


PI Materials 


Lessons 


1 


& 


2 


n 


None 


2 


20 sets ea. lesson 


Lessons 


3 


& 


4 




None 


2 


20 sets ea, lesson 


Lessons 


5-8 




H 


None 


2 


20 sets ea, lesson 




H 


is 


1 




Totals 








4 


25 


7 


20 sets ea. lesson 



Because so many students will be requiring audiovisual equipment at the 
beginning of lesson 2, before self •spacing has spread students among the les-^ 
sons, It was decided to locate all carrels in a room separate from that to 
be used for studying the PI materials. 

Figure C-2 shows planned student flow within Block IV, The student enters 
classroom j?l» He obtains the necessary materialr for that lesson. He begins 
studying his PI text until the lesson materials direct him to an audiovisual 
carrel > He then poes to room 92» If all the carrels are occupied, he either 
waits for one to become available or he takes a short break. Upon concluding 
his Interaction with the carrel, he returns to Ms original seat in room #1. 
This type of interaction may be continued until lesson terminus* Upon con- 
cluding the lesson, the criterion test is turned in to the Instructor for 
scorlngr If deficiencies are noted, a remedial prescription is generated by 
the lu jtructor and remediation is carried out in classroom 02b \then the 
studen ; has successfully completed a lesson, he turns in his materials and 
either 1) stays in the sarot room and obtains new lesson materials, or 2) moves 
to the room for lessons 3 and 4, 
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Learning time data collected clwrlng validation trials haa been used to estimate 
the times tequired for carrel use and to complete lessons within Block 1. 
Table C-3 presents this ln£onnatlon» In hours and minutes. 



Table C-3. Estimates of Learning Times 





Carrel 


^ Estimated Median 


Lessons 


time 


Lesson Time 


1 


« _ 


2:15 


2 


Ot38 


7:45 


3 


0:17 


4:15 


4 . 


0:17 


3:30 


5 


0:36 


3:30 


6 


0:22 


3:15 


7 




2:00 


8 




1:00 




1:30 


27:30 



Dally Scheduling Log 

A dally student log will be required for each Block IV student for allocation and 
scheduling on a dally basis. Figure G-3 presents a sample log. The log will 
aid the Instructors In ascertaining which resources will be used the next day, 
how a student is progressing, and where a student Is at any given tlme% The log 
should be designed and located so that the students themselves can ordinarily 
enter the required information. 

The instructors can review the log at the end of the day as a planning aid» 
Reserve media and materials, and can prepare for tutorial help or demonstrations 
required for the next day. The log book might be kept in the room in which 
the carrels are, Room 2, 
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Hiving the student £111 out the l6g hlmaelf plays a part In Incentive management 
Hiding in making the student feel responsible for his rate of progress* An 
ins tttictor will nead to monitor this activity rather closely at first to ensure 
that students do not set unrealistic or Irresponsible goals for themselves* 
Until empirical data on student rates of progressions are avallablei Instructors 
should not push students unduly to finish quickly. Validation studies completed 
shov that the maximum time required to finish Block IV should be less than a 
week* 



• Name 




• ID Humber 




9 cjvpectea course completion dace 




w iieoBon utoauxeo coapiQkeai inciuoing 
criterion tests pacified 




Day It 




0 Location now 




♦ Expected completion dat;e/time of current module 




• Expected need (next say) for: 




e tape/slide equipment 




• tape and slide sets (which) 




• new lesson module (which) 




• test monitoring and scoring 




e practical exercises or demonstration 
(which) 




^ Day 2t 




P' • Location now 





Figure Portions of a Daily Scheduling Log 



Stude nt/ I nstructor R oster 

The allocation and scheduling plan described In this plan will require a 
change to procedures for preparing a student/Instructor roster, since a given 
student will not have the same Instructor for a specific number of days. 

Inst ruct or/Student Load Balance 

Student load during the group-paced portion of the IM course can be handled much 
as it is currently. There should be very little problem vlth student distribution 
In Block IV classrooms because the projected median time for completion of Block 
IV is 27 hours 30 minutes — less than a week. Therefore, no new students will - 
be entering that block, since new students enter on a weekly basis, while the 
previous group is still present. Thus, the maximum number of students assigned 
to any given instructor would be 20, a load well within the instructor's 
capabilities to handle* 

Sc hedule Students for Job Assignment 

Because all students who enter the IM course in a given week will not be 
finishing on the same day, a procedure for scheduling students for early job 
assignment might be devised, A policy decision is required as to whether students 
who finish early will be given leave, kept at the school (if the time period is 
short), or scheduled for job assignment early. Early assignment could mean 
a change in Job location, the disadvantages of which must be considered. SDC 
feels that the solution to this problem is outside the scope of a technical plan* 
The problem will not be particularly severe for the IM course, because the self*- 
paced (final) block of instruction will be only a week along, which does not 
make it possible for a student fo finish the course more than a few days early« 
However, the IM course affords the opportunity to try several solutions in a' 
test bed environment to aid in selecting the best solution for the entire AIS 
at a later tiine« A change to personnel management policies with respect to 
job assignments is probably Indicated for courses that are entirely self-paced. 
Recommendations In this area are outside the scope of this Plan, but will 
require formulation for the operational AIS. 
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Conduct teaming Sees lone > Present Materials, ^gulpttent 



Individualization changes the instructor's role in the classroon* Instructors 
will require assistance in learning to perform this new role. The Instructor's 
Guides created along with the instructional materials are Intended to provide 
this assistance* SDC personnel will also brief the instructors on the materials 
and on the Instructor rolsi It is not thought that additional written 
assistance will be required. However, instructors should organize a 
workshop to familiarize themselves with the new materials and with their 
new roles. A suggested agenda is shown below. \ 

• The role of the instructor in an individualized 
instructional system 

• Student response to individualized instruction 

• Student motivation and incentive management 
e The new IM materials 

e The Instructor Guides 

e IM course organization and student flow 

e New procedures for instructors 

e The Student Report 

• The Scheduling Log 

• Scheduling and allocation o£ resources for the IM course 
e Administrative and evaluative activities 

e Dry run of Instructional activities ^ 

e Debriefing critique 

It la recommended that SDC personnel responsible for materials validation 
attend the workshop and participate in Its development. 



Monitor Student Progrees 

The InetroctorU Guide also serves as an aid in monitoring student progress |. 
as do the Scheduling Log and the Student Reports for planning. 

Furnish Tutorial Aidj Counselling Revised Plans and Schedules 

When students require tutorial aid, the instructor can provide this himself 
or select a capable student to provide the ald» A policy decision Is required 
as to the extent to which the capable student is to be utilized in this 
activity. An important consideration is the follpving': If students who 
finish early are to be rewarded by being given leave or passed i they may 
perceive tutoring as a punishment.^ In addition, if a particular student 
sets high goals for himself and is diverted from finishing by his Initially 
estimated completion date, he may also perceive this as punishment. A lack 
of. motivation, with respect to both course work and training, could be the 
result. On the other hand, capable students can function very effectively 
as tutors. It may be that, if policy decisions favor use of students as 
tutors and reject the concept of early Job adalgnment, a lack of emphasis on 
early completion dates would be desirable. In this case, the earliest 
estimated date of completion could be established as that for the average 
student. 

Administer Criterion Tests 
No changes required. 



147 



Score Tests t Analyze Student Performance 



Inatructlons for scoring criterion tests will appear in the instructor's 
Guide developed to accompany the new Instructional materials. Instructions 
for analysing student performance, on tests and also on instructional it^ms, 
MilX also be Included In the Guides. 

Determine Daily Diagnosis and Prescriptive Options 

Instructor's will need to monitor student performance and offer tutorial ald> 
counsel^ or other advice on a daily basis. The Instructor's Guides provided 
with the instructional materials will offer guidance on diagnosis of student 
problems and determination of useful prescriptions for the individual student. 
The Student Report progress suimnary can also be used as a decision-making aid* 

Update Student Records, Planning Reports 

Instructors can send student records Information to the administrative 
staff as is currently done* Student reports should be updated dally as a 
result of student-Instructor Interaction, instructor examination of student 
work, and test results. 

Monitor Resources Effectiveness and Student Performance 

School personnel whose activities are directed at course evaluation, student 
performance evaluatlon^and Instructional materials revision will l>ave more 
data available to them to use in these activities than is currently available » 
The Student Reports prepared by the instructor can be made available to them, 
as can the instructional materials used by the students. Students should 
therefore be encouraged to physically mark their answers to the programmed 
instruction items on the materials and to strike through incorrect answers 
rather than erase them. Other data desired by the evaluation staff should 
be Identified and collected by the evaluation staff Itself during the manual 
phase, to avoid overloading Instructors with duties while they are learning 
their new roles. 
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Re duce and Analyge Performance Data> Analyze Course Materials > 
Analyze Job> Decide Revlslona 

Some amount of change In orientation of analysis activities will be required 
by lndlvlduallZ£(tlon of portions of the IM coursei particularly for analysis 
of course materials. Analysis can be directed at discovering which Instructional 
frames I If any^ are falling to teach satisfactorily. In the traditional course/ 
this level of detail Is usually not possible for course analysis and evaluation. 
The analysis staff can develop procedures for rllecovering and revising fra^aes 
'that do not meet established effectiveness criteria. Part Two of this 
report can be used as a guide to job analysis and development of materials 
should the job change or Segments of the materials prove Ineffective. 

Develop/Revise Task Data» Objectives, Instructional Hater lals> test Items>^ 
Prescriptive Rules, Instructor Support Materials 

Instructional materials revision can be accomplished on an item by item 
basis as required. Part I^o of this Technical Report can serve as a guide 
to preparation of new or revised instructional f^amesi test items> job 
data or objectives. Prescriptive rules and Instructor support materials 
can best be revised as a result of ongoing instructor-evaluator coordination* 
A series of instructor-evaluator workshops is recommendedt in which 
evaluators provide positive guidance on rules and materials modif lcatlon» 
based on evaluritlon data and upon instructor experiences in the classroom* 

Prepare and Output Evaluation Results Reports 
No changes required. 
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VALIDATION STUDY AND DATA 
APPENDIX D. 



INTRODUCTION 

The purposes. of the Air Force tryouts were to: (I) verify 
the revised materials? and (2) to obtain additional data necessary 
to plan for a full scale Implementation of the materials* The re** 
vised block IV materials were administered to a total of 57 students 
In three different groups by Air Force Instructor personnel. Most 
of the 57 students were selected from a single shift (C) and Inadvert* 
ently volunteers were permitted* 

Data were gathered for each student on learning time per module > 
days to completion i end of block test score and aptitude scores. 
Student critiques were obtained anonymously at the end of the block 
from each student* 

The tryout team consisted of three Instructors. The tryouts 
were conducted In three rooms. One room was used for training and the 
other two for testing. (The second test room was used for block testing 
Initially, but this practice was discontinued when It became evident 
that one room provided adequate test security.) Figure D-1 shows the 
layout of the training roomi 

The Instructor oriented the students about the materials and pro 
cedures to be used and then permitted them to proceed. Figure D-2 Is 
a decision logic chart of the process In this tryout. As each student 
finished a module^ he was tested. Those who did not pass a module were 
given remedial training and were retested. At the completion of module 
eighty each student took the block test and was graduated. Had there 
been any failures, a washback or remedial process would have, been used. 
Each student completed the critique after passing the block. 



ANALYSIS and FINDINGS 

The 57 students completed the block IV material In 45% of the 
normal block time. They achieved a mean block test score of 88.6%. 
All students passed the block test on the first try. Background and 
end of block teat data are given in Table D-I. The distribution of days 
to completion of Block IV is given in Table D-2. 

The 57 students achieved a mean time savings of 55%. This was 
somewhat better than that attained by the 20 students in the formative 
evaluation. The difference may be due to the improvements In the learn- 
ing materials, the added experience of the instructors in managing the 
system, and;or differences among student abilities. 
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The critique data are given in Table D*3. The reactions of 
the students were highly favorable toward self-pacing and they found 
the materials easy to read. The reaction to the audiovisual material 
was quite favorable, but there were indications that versions of the 
audlovisuals used for review should be different irora those used for 
advanced organizers. Because some of the student^ objected to using 
the same audiovisuals twice > the last group of students were given 
the option of skipping these in review. Only two of 16 students chose 
to akip the review. 

Background data were obtained on an additional group of 100 
students who were conventionally trained in Block IV in nearly the 
same time frame as the 57. The data for the 57 validation students, 
the 20 students in the formative evaluation, and the 100 other "students 
were compared with respect to AQE and prior block scores. Table D-4 
gives the means and standard deviations for the AQE[8 and prior block 
grades for these groups. The 20 formative evaluation students and the 
100 conventionally trained students do not differ markedly. If any- 
thing, the 20 have lower scores. The 57 validation students do have 
highipr block scores and higher general and administrative qualification 
examination scores. The higher course scores probably have more meaning 
thaa the AQE scores and probably account for part of their increase in 
performance nver the formative group. 

The block IV scores and the learning times for the 57 validation 
students and the 20 students in the formative evaluation were compared. 
The validation group achieved higher scores in less time. Because the 
validation group had a better prior academic record as well as improved 
learning materials, the gains could be attributed to either the improve- 
aent of the materials or the willingness and ability of the individuals 
to learn. It is probably reasonable to expect that tiean scores and 
times for large groups of inventory management students be between 
those for the 20 formative evaluation students and the 57 validation 
students. 

The validation made use of separate rooms for training and testing. 
The instructors had no difficulty with this arrangement. In the learning 
room the instructor activity was more concentrated in the initial orienta- 
tion and in interactions with students during the first two modules. This 
was partly to acquaint the students with the system and the importance of 
interacting with all of the materials. One instructor handled a room of 
20 students with ease and indicated that a somewhat larger group size 
might be possible. 
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The activity level In the test room was such that one instructor 
could test students from two learning ropms* Thus three instructors 
could accommodate 40+ students without difficulty provided that they 
were given some relief for breaks* This instructor student ratio is 
no greater than conventional instruct; ion. 

There would appear to be no reason why this system could not 
be employed on a large scale if adequate audiovisual materials are 
provided. The precise layout of rooms and audiovisual devices would be 
a function of facility availability. Under some circumstances it might 
be preferable to locate audiovisual devices in one location, but it would 
appear that decentralized audiovisual devices is better » 
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Classroom Configuration 
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Tabl6 


D-1. 


Background and End of Block Test Data 






Gen. 




Ifech. 


Elect. 


AFQT- ' 


Block 
Test 


1 


60 


■ 60 


45 


45 


33 


92 


2 


60 


85 


60 


65 


3S 


94 


3 . 


95 


80 


30 


65 


54 


96 


4 


70 


65 


50 


65 


36 


90 


5 


80 


70 


90 


90 


91 


86 


6 


85 


75 


70 


60 


80 


96 


7 


55 


80 


40 


20 


60 


92 


8 


65 


80 


45 


65 


82 


80 


9 


75 


70 


55 


80 


78 


92 


10 


80 


60 


65 


70 


60 


96 


11 


60 


95 


45 


55 


38 


88 


12 


45 


95 


35 


40 


36 


90 


13 


25 


60 


40 


40 


42 


92 


14 


60 


40 


65 


60 


76 


84 


15 


75 


70 


45 


55 


49 


94 


16 


65 


80 


80 


50 


48 


92 


17 


50 


60 






J** 


OO 


18 


50 


80 


30 


65 


58 


88 


19 


55 
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TABLE D - 2 



DAYS TO BLOCK 
COMPLETION 

3 

4 

5 

6 

7 



NUMBER OF STUDENTS 
COMPLETING BLOCK 

2 
28 
23 

3 

1 



TABLE D - 3 

QUESTION YES (?) NO 

1, Do you like self -pacing? 54 1 2 

2» Did r.iaterial cover course content? 57 

3. Was the material easy to read? 55 2 

4. Were the tape-slides useful? 

a. Before 40 5 12 

b. During 42 5 10 

c. After 39 4 14 

5. Did you learn better with this 

method? 43 6 8* 

*rhe comments of- six of these persons indicated they equated self-pacing 
with group pacing" although their check mark was in the no column. 
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STAGE 3 VALirJATION 
STUDENT LEAFNIl^G TIMES 
LESSONS 1-8, BLOCK IV 
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STAGE 2 VALIDATION 
STUDENT PERFORMANCE SCORES j 
LESSONS 1-8, BLOCK IV 
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TOTAL 


Lesson 

Mean 

Score 




0, 

Torvg 


% 

Correct 


No. 
Wrong 


Correct 


Student 


Wronp 


Mean X 
Correct 


1 


N/A 


■ N/A 


6 
6 

' 2 
5 


71 
70 
90 
76 


1 
2 
3 
4 


6 
6 
2 
5 


■ 71 
70 
90 
76 


77 




Item N- 






21 


21 


2 


9 
5 
2 
9 


67 
81 
93 
67 


12 
14 
6 

12 


65 
59 
82 
65 


1 
2 
3 
4 

5 
6 
7 
8 

5 
6 
7 
8 

5 
6 
7 
8 

5 
6 
7 
8 

5 
6 
7 
8 

5 
6 
7 
8 


21 
19 
8 

21 


66 
69 
87 
66 


72 




3 

Item N- 
4 

Item N- 
5 

Item N- 
6 

Item N- 
7 

Item N" 
8 

Item N- 


2 


7 


3 


4 


61 


3 
6 
4 


^1 

87 • 

74 

83 


2 
5 
5 
5 


93 
83 
83 
83 


4 
8 
11 
9 


92.5 ■■ 
85 
79 
83 


85 




23 


3 


0 


53 


4 
3 
4 

5 


78 
83 
78 
72 


2 
0 
0 
1 


91 
100 
100 

96 


6 
3 
4 
6 


85 
93 
90 
85 


88 


16 


23 


41 




4 
7 
8 

4 


86 
76 
72 
86 


0 
0 
0 
1 


100 
100 
100 
96 


4 
7 
8 
5 


93 
87 
85 
91 


89 




29 


26 


55 


13 1 
10 
16 
13 


64 
72 
56 
64 


2 
2 
1 
7 


95 
95 
97 
82 


15 
12 
17 

.20 


80 
84 
77 
73 


79 . 




36 


39 


75 


4 
4 
6 
4 


80 
80 
70 
80 


4 
5 
4 
6 


64 
55 
64 
45 


8 
9 
10 
10 


74 
71 
68 
68 


70 






11 




1 


0 
0 
0 
0 


100 
100 
100 
100 


4 
2 
0 
4 


67 
83 
100 
67 


4 
2 
0 
4 


79 

89.5 
100 
79 


87 




7 


12 






.9 



ERIC 



Example 
170 



ENABLING 
CRtTERlON X 



IV 



TEST ITEM MALVSIS 



LESSON 
3 4 



BLOCK 
5 



10 



n 



128 



21 








ALLOWANCE 


EA 

AUT^IORITY 


AUTHORITY 
NO APPROVi^ 


L ^ 


22 


TV 

u 








E AUTHORI- 
ZATION AC! 


Y (BLANK) 




23 










EA 

AUTHORITY 


ACTIVITY 
NO APPROV/ 


L ' 






(BLANK) 


(BLANK) 


(BLANK) 






I 

J 




12C 


12D 


12E 


TOTALS 
WRONG % 




21 




E 


C 


6 


71 




1 


1 

22 


E 




B 


6 


70 






23 








2 


90 




1 


24 


E 




B 


5 


— 




1 












%¥■ 







ERIC 



Example 



171 



-J 



EHABLINQ 
CRITERION 



8 



IV 



TEST ITEM ANALYSIS 



LESSON 



BLOCK 



« 


WRONG 


% 


25 


0 


100 






■ 1 






• 26 


0 


100 












27 


0 


100 












28 


0 


100 











































i 

















































1 
1 
















! 



































Example 



ENABLING 

CRITERION X 



6 IV 

LESSON - ■ BLOCK 



TEST ITEM ANALYSIS 



3 6 7 - 8 9 WRONG . X 



25 


E 




WRM 
ITEMS 


D ONLY , 


B ONLY 


4 


67 


26 


E 






C ONLY 




2 


83.5 


27 












0 


100 


• 28 


E 


B 




BC 


E 


4 


67 

















1 


































1 

































































i 

















Example 



STAGE 3 VALIDATION 
STUDENT PERFORMANCE SCX)RES i 
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APPENDIX E 
COST ANALYSIS REPORT 



COST AMALYSIS REPORT 

SDC wd8 ashed to provide IM project cost data and to estimate potential 
reductions In costs for similar projects performed In a production 
configuration. 

COST DATA 

Man-hour data were extracted from project records for the IM course cost 
analysis* Direct man-hours were as follows for the project: \ 
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Direct Man 
Hours 



PHASE I 



Analysis of IM course 104 
Selection of course segments 37 
Prepare Technical Report 34 

175 

PHASE II 

Task Analysis > 332 

Develop Learning Objectives 417 

Develop Criterion Tests 156 

Develop Media Selection Strategy 33 

Develop Instructional Strategy 25 

Prepare Technical Report 475 

1,126 

PHASE XZZ 

Develop Instructional Materials 1^668 
Perform Pilot Study 210 
Develop Management Strategy 63 
Prepare Technical Report 103 

2^044 

PHASE IV 

Evaluation & Revision 394 
Prepare Technical Report 211 

605 

Total Man- Hours 3,950 



Materials and related costs were as follows for the project (in dollars): 

Dollars 



PHASE I 



Editorial Support 
Reproduction 
TDY Travel 



37 
33 
493 

563 
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Dollars 



76 
107 
246 

429 
PHASE III 



Editorial Support i 980 

Reproduction - 13 Lesson Modules 173 

TDY Travel 402 

Equipuent (Uollensak Cassette Recorder) 240 

Art Supplies & Tape Cassettes 286 

Slide Development (545 - 35inin) 3.501 

* 5,582 

PHASE XV 

Editorial Support 729 

Reproduction 2,874 

TDY Travel 2,614 

Black Slide Trays (180) 324 

35inm Duplications of 545 (29 sets) 4,794 

'Visual Arts 340 

Tape Cassettes Duplications of 11 (29 sets) 1.058 

"I *■-■'■ 



12,733 

Total Dollars 19,307 



PHASE II 

Editorial Support 
Reproduction 
TDY Travel 



PRODUCTION CONFIGURATION ESTIMATION RATIONALE 

SDC was aaked to estimate the reduction in costs that might occur if a project 
such as the IM project was performed in a production configuration rather 
than In a research and development configuration. A rationale to account 
for cost differences was therefore prepared. Because SDC did not manage the 
IM project in a true RiD configuration , being production-oriented to a con- 
siderable degree, two sets of cost reduction assu:nptions Uere prepared for 
the estimation task. The first set of assumptions would be appropriate for 
estimating production configuration cost reductions from R&D configuration 
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costs. The second I which was actually used to make estimations for this 
report I Is appropriate for estimating production configuration cost reduc- 
tions from SDC's cost records. The three types of configurations can be 
compared as shown in Table E-l. 



Table B-1. Configuration Comparison 



Comparison Factors 


R&D 

Configuration 


SDC's 
Configuration 


Production 
ConflgutHtion 


Amount of specialization 
or project personnel 


Either Very 
Low or Very 
High 


Medium 


High 


Diversity of skill and 
salary levels of project 
personnel 


Low 
Low 


Medium 
Medium 


High 
High 


Amount of in*house 
production equipment 
utilized on project 


Low 


Medium 


High 


Volume of related 
business ongoing i over 
which project person- 
nel i equipment and 
facilities are shared 


Low 


Medium 


High 


Proportion of method- 
ologies t rationales » 
and strategies required 
to be developed espe- 
cially for project 
work 


High 


Medium 


Low 


Amount of Job aids 
available to guide 
project personnel 

k 


Low 


High 


High 



SDC believes that cost reductions would be considerable In a changeover from 
an R&D configuration to a production conf lguratlon» given that the following 
assumptions holdi 



• In an R&D conf Iguratloni a major manpower-related cost is associated 
with design and development of special methodologies » strategies and 
rationales for producing project outputs. An example would be devel- 
opment of a method for selecting methods and media for each lesson 
element. In a production conf lguratlon» existing methodology Is 
merely adapted for use In a specific project^ and job alds^ such as 
forms I checklists, and handbooks are available, reducing direct man- i 
hour costs by an estimated 20%. 

# In an R&D configuration, principal researchers, with excellent academic 
credentials and years of research experience, often perform the bulk 

of the project work themselves, farming production work out-of-house 
and perhaps using relatively Inexperienced personnel for the simplest 
tasks# In a production configuration, each project member Is special- 
ized and his skill and salary level Is appropriate to the sulset of 
tasks he performs. . All project tasks are done In-house, utilizing 
In-house personnel^ equipment and facllltlesi the costs of which are 
shared with other similar projects. Estimated man-hour cost reductions 
In the production configuration are 10%, due to 1) a reduction In the 
number of high skill and salary level personnel required, 2) the some- 
what lower total number of hours required for specialized production 
personnel to perform lower level tasks In comparison with use of In- 
experienced personnel or students, and 3) the work-load balancing 
made possible by a high volume of related projects. Estimated cost 
reductions for materials In the production configuration are 2SZ, 
based primarily on the fact that audiovisual materials processing 
and reproduction costs are less In the production configuration, 
being Incurred In-housei and also that equipment costs are shared 
over many similar projects. 



Cost reductions In a changeover from SDC's production-oriented configuration 
to a true production configuration are less than can be obtained In a change- 
over from a true R&D configuration. The following assumptions must be used 
In making cost reduction estimations: 
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4 Mo Qan«-hour costs can be attributed to design of special methods'>logie8 
for performing project tasks i since existing methodologies were used 
instead I as they would be in a production configuration. Costs of 
adaptation of methodologies » strategies » and rationales for the IH 
project were about the same as would be Incurred in a true production 
configuration. 

j» Reductions in man-hour costs due to the possibilities for increased 
puteonnel speclallasation and diversity of skill and salary levels In 
a production configuration are small » because SDC has adopted a pro« 
ductlon«"oriented stance toward manning of projects like the IM project* 

0 Instructional materials production costs would be somewhat less In a 
true production configuration with a high volume of related work« It 
is estimated that as much as a 25% saving would be possiblei due to 
differences in volume-*buying costs of equipment and material and to 
methods of accounting for production and processing charges. 



ESTIHATBS OP PEDUCTIONS IN COSTS 

Estimations of reductions in costs for specific items ^re listed in Table P-2f 
which applies to cost differences in the SDC configuration versus a production 
configuration. Percentages represent reductions in dollar costs throughout 
the table. ^ 



Table E-2. 



Cost Reduction Estimates 



Cost Element 



Reduction in Production 
Configuration 



Phase I 

Analysis of IM Course 
Selection of Course Segments 
Prepare Technical Report 
Editorial Support 
Reproduction ^ 



OX 

o; 

0% 
OX 
lOX 
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Cost Element 






Phase II 




• 








Develop Learning Objectives 


0% 




Develop Criterion Tests 


0% 




Develop Media Selection Strategy 


0% 




Develop Instructional Strategy 


ox 




Prepare Technical Report 


60% 


(part two not required in pro- 






duction configuration) 


Editorial Support 


5% 


(tech* report shorter) 


Reproduction 


20% 


(tech* report shorter) 


TOY Travel 


100% 


(facility located at school) 

• 


Phase III 






ueveiop iuSuruCkionoi naueriais 




^inore specialization/ 


Perfortt Pilot Study 


5% 


(lover level personnel) 


Develop Management Strategy 


0% 




Prepare Technical Report 


20% 


(part two not required) 


Editorial Support 


5% 


(tech. report shorter) 


Reproduction 


20% 




TDY Travel 


100% 


(if facility located at school) 


Equipment (Cassette Recorder) 


90% 


(cost shared over several projects) 


Art Supplies and Cassettes 


10% 


(high volume buying) 


Slide Development 


25% 




Phase IV 






Evaluation and Revision 


10% 


(lower level personnel) 


PfADArA T^f^hnif^Al Renoirt 


50% 


^DAirtl two not IfAflul t*Ait^ 


Editorial Support 


5% 


(tech. >:eport shorter) 


Reproduction 


20% 




TDY Travel 


100% 


(if facility located at school) 


Black Slide Trays (180) 


10% 


(high volume buying) 


Duplication of Slides 


25% 




Visual Arts 


20% 




Duplication of Tape Cassettes 


30% 





The estimates shown in Tattle E-2 would result in an overall savlAg of f 

about 19% in dollars, if nb travel were necessary and if Par'* Two of the I 

Technical Report were not required. If these two cost elements were required » < 

then coat savings would be only 7-1/2%. ^ ^ | 
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